pith. sign in

arxiv: 2606.03788 · v1 · pith:Y44UJOE7new · submitted 2026-06-02 · 💻 cs.CV

SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation

Zeno Testa , Antonino Furnari , Lorenzo Baraldi , Natalia D\'iaz-Rodr\'iguez This is my paper

Pith reviewed 2026-06-28 10:48 UTC · model grok-4.3

classification 💻 cs.CV
keywords sign language translationsemantic evaluationquestion answering benchmarkmultimodal large language modelsPHOENIX-2014TCSL-Dailyvideo understanding
0
0 comments X

The pith

Sign language translation systems still miss key semantic details even after fine-tuning, scoring 56.7 to 75.2 percent on a new question benchmark.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shifts evaluation of sign language translation from surface metrics like BLEU to direct tests of semantic understanding. It creates SLU-2K, a set of 2,350 video question-answer pairs drawn from two existing datasets and generated across seven categories of meaning. Testing shows multimodal large language models perform near random while two representative state-of-the-art translation systems reach only 56.7 to 75.2 percent accuracy. These results indicate that current protocols based on lexical overlap overestimate how well translations preserve the original meaning. The work therefore argues that future progress must be judged by semantic correctness as well as fluency.

Core claim

The central claim is that state-of-the-art sign language translation systems fine-tuned on in-domain data still exhibit a substantial semantic gap on SLU-2K, reaching only 56.7 to 75.2 percent accuracy, while multimodal large language models reach near-random performance; this demonstrates that surface-form metrics overestimate true semantic understanding and that future SLT evaluation should incorporate semantic correctness.

What carries the argument

SLU-2K, a dataset of 2,350 closed-ended video question-answer pairs produced by an automated pipeline across the seven categories of actions, locations, numbers, objects, people, time, and weather conditions.

If this is right

  • Current SLT evaluation protocols overestimate true understanding because they rely only on fluency and n-gram overlap.
  • Future progress in sign language translation should be measured by semantic correctness in addition to existing surface metrics.
  • Systematic integration of semantic understanding evaluation is required in current AI systems for sign language.
  • Automated pipelines can generate large-scale question sets for semantic evaluation from existing translation datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same question-based approach could be extended to additional sign language corpora beyond the two source datasets used here.
  • Models may be exploiting dataset-specific patterns rather than learning general semantic mappings, which would explain the gap between translation fluency and question-answering accuracy.
  • Pairing SLU-2K scores with traditional metrics on the same outputs would give a more complete picture of model behavior.

Load-bearing premise

The automated question-generation pipeline produces questions that accurately and without bias reflect the semantic content of the original sign videos across the seven categories.

What would settle it

A study in which independent human raters judge a random sample of the generated questions as failing to capture the actual semantic content of their source videos would falsify the benchmark's validity.

Figures

Figures reproduced from arXiv: 2606.03788 by Antonino Furnari, Lorenzo Baraldi, Natalia D\'iaz-Rodr\'iguez, Zeno Testa.

Figure 1
Figure 1. Figure 1: The proposed Sign Language Understanding SLU-2K [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the SLU-2K Benchmark construction pipeline. A question and its distractors are first generated from a reference [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Benchmark composition across the main stages of data [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the model evaluation pipeline. Given an input sign video, an SLT model produces a predicted translation, which [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
read the original abstract

Sign Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directly measure whether a translation preserves the meaning of the source sign sequence. This is in contrast with the final objective of integrating SLT in assistive technology. In this work, we shift the focus from Sign Language Translation (SLT) to Sign Language Understanding (SLU), with particular emphasis on semantic understanding. Specifically, we evaluate systems based on their ability to correctly recover, from the input video, key semantic aspects of the original sentence, such as actions taking place and facts about people and objects. To enable this evaluation systematically, we propose SLU-2K, a dataset of 2,350 closed-ended video question-answer pairs based on the popular PHOENIX-2014T and CSL-Daily datasets. To obtain SLU-2K, we propose and extensively evaluate an automated data generation pipeline which produces questions across 7 categories, namely actions, locations, numbers, objects, people, time, and weather conditions. We show the potential of SLU-2K by evaluating popular Multimodal Large Language Models (MLLMs) and two representative state-of-the-art systems, MMSTL and SpaMo. Our results show that MLLMs reach near-random performance, highlighting the need for a more systematic integration of SLU in current AI systems. Furthermore, state-of-the-art translation systems carefully fine-tuned on in-domain data still exhibit a substantial semantic gap, with results ranging from 56.7% to 75.2%. These findings suggest that current SLT evaluation protocols overestimate true understanding and that future progress should be measured not only by fluency and n-gram overlap, but also by semantic correctness. Code, prompts, and benchmark files are available at https://github.com/ZenoTsT/SLU-2K

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces SLU-2K, a benchmark of 2,350 closed-ended video question-answer pairs derived from PHOENIX-2014T and CSL-Daily via an automated pipeline across seven semantic categories (actions, locations, numbers, objects, people, time, weather). It evaluates MLLMs and two SOTA SLT systems (MMSTL, SpaMo), reporting near-random MLLM performance and SOTA accuracies of 56.7%–75.2%, and argues that surface metrics like BLEU overestimate semantic understanding in SLT. Code, prompts, and benchmark files are released publicly.

Significance. If the automated pipeline produces questions that faithfully and without bias capture semantic content directly from the sign videos, SLU-2K would offer a useful complement to n-gram metrics by directly testing meaning preservation, a key requirement for assistive SLT applications. The public release of code and data is a clear strength supporting reproducibility.

major comments (2)
  1. [automated data generation pipeline] Abstract and automated data generation pipeline section: the claim that the pipeline was 'extensively evaluated' is not supported by any reported quantitative validation (e.g., inter-annotator agreement, fraction of questions verified answerable from video alone, or error analysis for artifacts); this directly affects the reliability of the headline accuracy figures (56.7%–75.2%).
  2. [evaluation section] Abstract and evaluation section: it is not stated whether question generation begins from video content, glosses, or text translations; if the latter, the benchmark cannot be guaranteed to test video-based semantic recovery, which is the central premise for evaluating SLT systems on semantic gap.
minor comments (2)
  1. [Abstract] Abstract: the range 56.7% to 75.2% should be broken down by system (MMSTL vs. SpaMo) for precise interpretation of results.
  2. References: ensure full citations are provided for MMSTL and SpaMo, which are described as representative SOTA systems.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive feedback. We address each major comment below and will revise the manuscript to improve the description of the data generation pipeline and its validation.

read point-by-point responses

  1. Referee: Abstract and automated data generation pipeline section: the claim that the pipeline was 'extensively evaluated' is not supported by any reported quantitative validation (e.g., inter-annotator agreement, fraction of questions verified answerable from video alone, or error analysis for artifacts); this directly affects the reliability of the headline accuracy figures (56.7%–75.2%).

    Authors: We agree that the current manuscript does not provide quantitative validation metrics for the pipeline, such as inter-annotator agreement or explicit error rates on answerability from video. The description relies on qualitative checks and manual inspection of a subset of outputs. In the revised version, we will add a new subsection in the data generation section reporting quantitative results, including the fraction of questions verified as answerable from video alone and a systematic error analysis of artifacts. revision: yes

  2. Referee: Abstract and evaluation section: it is not stated whether question generation begins from video content, glosses, or text translations; if the latter, the benchmark cannot be guaranteed to test video-based semantic recovery, which is the central premise for evaluating SLT systems on semantic gap.

    Authors: The pipeline generates questions from the text translations in the source datasets (PHOENIX-2014T and CSL-Daily), which are time-aligned with the videos and glosses. Questions are designed to target semantic content that is visually present in the sign videos. We acknowledge that the starting point of generation is not explicitly stated in the current text. In the revision, we will add a clear description of the pipeline steps, including examples, and explain how the resulting questions still evaluate video-based semantic recovery when applied to MLLMs and SLT systems. revision: yes

Circularity Check

0 steps flagged

No circularity; benchmark and pipeline are externally constructed and evaluated

full rationale

The paper introduces SLU-2K as a new benchmark of 2,350 video QA pairs generated via an automated pipeline from public datasets (PHOENIX-2014T, CSL-Daily) across 7 categories, then reports empirical results on MLLMs and SLT systems (MMSTL, SpaMo). No equations, fitted parameters, predictions, or derivations appear. Central claims rest on direct evaluation of existing models against the new benchmark rather than any self-referential reduction, self-citation load-bearing step, or ansatz. Matches reader's assessment of score ~1 with no load-bearing circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The work introduces no free parameters, no new axioms beyond standard assumptions about video captioning and question generation, and no invented entities; it re-uses two existing public datasets and adds a new evaluation layer on top.

pith-pipeline@v0.9.1-grok · 5893 in / 1195 out tokens · 22477 ms · 2026-06-28T10:48:25.418753+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 6 canonical work pages

  1. [1]

    Hierarchical feature alignment for gloss-free sign language translation

    Sobhan Asasi, Mohamed Ilyes Lakhal, and Richard Bowden. Hierarchical feature alignment for gloss-free sign language translation. InAdjunct Proceedings of the 25th ACM Inter- national Conference on Intelligent Virtual Agents, pages 1–6,

  2. [2]

    Beyond gloss: A hand-centric framework for gloss-free sign language translation.arXiv preprint arXiv:2507.23575, 2025

    Sobhan Asasi, Mohamed Ilyas Lakhal, Ozge Mercanoglu Sincan, and Richard Bowden. Beyond gloss: A hand-centric framework for gloss-free sign language translation.arXiv preprint arXiv:2507.23575, 2025. 1, 2

    work page arXiv 2025

  3. [3]

    Neural sign language trans- lation

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Her- mann Ney, and Richard Bowden. Neural sign language trans- lation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7784–7793, 2018. 2, 3

    2018

  4. [4]

    Factorized learn- ing assisted with large language model for gloss-free sign language translation

    Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. Factorized learn- ing assisted with large language model for gloss-free sign language translation. InProceedings of the 2024 Joint In- ternational Conference on Computational Linguistics, Lan- guage Resources and Evaluation (LREC-COLING 2024), pages 7071–7081, 2024. 1, 2

    2024

  5. [5]

    C2RL: Content and context representation learning for gloss- free sign language translation and retrieval.IEEE Transac- tions on Circuits and Systems for Video Technology, 2025

    Zhigang Chen, Benjia Zhou, Yiqing Huang, Jun Wan, Yibo Hu, Hailin Shi, Yanyan Liang, Zhen Lei, and Du Zhang. C2RL: Content and context representation learning for gloss- free sign language translation and retrieval.IEEE Transac- tions on Circuits and Systems for Video Technology, 2025

    2025

  6. [6]

    LLMs are good sign language translators

    Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. LLMs are good sign language translators. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18362–18372, 2024. 1, 2

    2024

  7. [7]

    Q2: Evaluating fac- tual consistency in knowledge-grounded dialogues via ques- tion generation and question answering

    Or Honovich, Leshem Choshen, Roee Aharoni, Ella Nee- man, Idan Szpektor, and Omri Abend. Q2: Evaluating fac- tual consistency in knowledge-grounded dialogues via ques- tion generation and question answering. InProceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7856–7870, 2021. 3

    2021

  8. [8]

    True: Re-evaluating factual consistency evaluation

    Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitel- baum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re-evaluating factual consistency evaluation. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technolog...

    2022

  9. [9]

    An efficient sign language translation using spa- tial configuration and motion dynamics with LLMs.arXiv preprint arXiv:2408.10593, 2024

    Eui Jun Hwang, Sukmin Cho, Junmyeong Lee, and Jong C Park. An efficient sign language translation using spa- tial configuration and motion dynamics with LLMs.arXiv preprint arXiv:2408.10593, 2024. 1, 2

    work page arXiv 2024

  10. [10]

    A spatio-temporal representation learn- ing as an alternative to traditional glosses in sign language translation and production

    Eui Jun Hwang, Sukmin Cho, Huije Lee, Youngwoo Yoon, and Jong C Park. A spatio-temporal representation learn- ing as an alternative to traditional glosses in sign language translation and production. In2025 IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV), pages 3352–3362. IEEE, 2025. 2

    2025

  11. [11]

    Lost in translation, found in context: Sign language translation with contextual cues

    Youngjoon Jang, Haran Raajesh, Liliane Momeni, G¨ul Varol, and Andrew Zisserman. Lost in translation, found in context: Sign language translation with contextual cues. InProceed- ings of the Computer Vision and Pattern Recognition Con- ference, pages 8742–8752, 2025. 2

    2025

  12. [12]

    Leveraging the power of mllms for gloss-free sign language translation

    Jungeun Kim, Hyeongwoo Jeon, Jongseong Bae, and Ha Young Kim. Leveraging the power of mllms for gloss-free sign language translation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21048–21058, 2025. 1, 2

    2025

  13. [13]

    Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods.arXiv preprint arXiv:2203.05227, 2022

    Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods.arXiv preprint arXiv:2203.05227, 2022. 3

    work page arXiv 2022

  14. [14]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004. 3

    2004

  15. [15]

    Spotter+ GPT: Turning sign spottings into sentences with LLMs

    Ozge Mercanoglu Sincan and Richard Bowden. Spotter+ GPT: Turning sign spottings into sentences with LLMs. In Adjunct Proceedings of the 25th ACM International Confer- ence on Intelligent Virtual Agents, pages 1–6, 2025. 2

    2025

  16. [16]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318,

  17. [17]

    Multilin- gual gloss-free sign language translation: Towards building a sign language foundation model

    Sihan Tan, Taro Miyazaki, and Kazuhiro Nakadai. Multilin- gual gloss-free sign language translation: Towards building a sign language foundation model. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 553–561, 2025. 1

    2025

  18. [18]

    Stochastic transformer networks with linear competing units: Application to end-to-end sl translation

    Andreas V oskou, Konstantinos P Panousis, Dimitrios Kos- mopoulos, Dimitris N Metaxas, and Sotirios Chatzis. Stochastic transformer networks with linear competing units: Application to end-to-end sl translation. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 11946–11955, 2021

    2021

  19. [19]

    Sign2GPT: Leveraging large language models for gloss-free sign language translation.arXiv preprint arXiv:2405.04164, 2024

    Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2GPT: Leveraging large language models for gloss-free sign language translation.arXiv preprint arXiv:2405.04164,

    work page arXiv

  20. [20]

    Towards inclusive communication: A unified framework for generating spoken language from sign, lip, and audio.arXiv preprint arXiv:2508.20476, 2025

    Jeong Hun Yeo, Hyeongseop Rha, Sungjune Park, Junil Won, and Yong Man Ro. Towards inclusive communication: A unified framework for generating spoken language from sign, lip, and audio.arXiv preprint arXiv:2508.20476, 2025. 2

    work page arXiv 2025

  21. [21]

    SLTUNET: A simple unified model for sign language translation.arXiv preprint arXiv:2305.01778, 2023

    Biao Zhang, Mathias M ¨uller, and Rico Sennrich. SLTUNET: A simple unified model for sign language translation.arXiv preprint arXiv:2305.01778, 2023. 2

    work page arXiv 2023

  22. [22]

    Gloss-free sign language translation: Improving from visual- language pretraining

    Benjia Zhou, Zhigang Chen, Albert Clap ´es, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual- language pretraining. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 20871– 20881, 2023. 1

    2023

  23. [23]

    Improving sign language translation with monolingual data by sign back-translation

    Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1316–1325, 2021. 2, 3

    2021