{"id":"e6604f6c-bad7-4aa4-bf1f-66ce008faa22","arxiv_id":"2501.09754","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Feeding an LLM sign-language visual features plus pseudo-glosses, previous-sentence translations, and background captions improves open-vocabulary sign language translation on BOBSL and How2Sign.","lead":"This paper builds a sign language translation system that feeds an LLM not only the signing video but also background scene descriptions and the translation of the previous sentence. This context helps resolve ambiguities like homonyms, pointing, and pronouns, and improves translation quality on the large BOBSL dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing text-only baseline leaves the marginal contribution of visual features to the full-cue model unquantified; the claim of significant context-augmented SLT improvement over video-only is confounded by LLM text priors.","rationale":"The reader's weakest assumption is that the LLM may rely on text cues rather than visual features, and the missing text-only baseline is exactly the load-bearing gap. I agree with the reader's assessment. The paper's ablation tables only remove cues from a video-present base; they never remove video while keeping text cues. Given the LLM is a strong language model and previous-sentence translations are fluent English, the model could generate plausible translations from text context alone. The paper does demonstrate robustness via Drop Cue augmentation (Tab. A.6) and shows that a video-only model beats GFSLT and Sign2GPT, which supports a genuinely strong visual encoder. But that does not establish the marginal contribution of video in the full-cue setting. The conditional verdict is appropriate: the central claim is plausible and well-supported in several places (e.g., Table A.9 shows the ISLR backbone is strong; the human-correlation of the LLM metric in Fig. A.2 is real evidence), but the missing no-video baseline is a concrete, easily addressable gap. I would keep the verdict CONDITIONAL and require the text-only ablation before full acceptance. This is not a rejection because the paper's own reported video-only performance (BLEURT 37.8 vs full-cue 40.3) suggests video alone already does well; the question is whether the full-cue model is genuinely multimodal or effectively text-only with a video token appended.","tokens_in":26902,"tokens_out":1896,"duration_ms":16464,"concrete_test":"Run the full-cue model with the visual feature channel removed: input only pseudo-glosses, predicted previous sentence, and background description (PrevPred+PG+BG, no Vid tokens) on BOBSL SENT-VAL and SENT-TEST. If this text-only model matches or nearly matches the full-cue scores (e.g., BLEURT within ~1 point of 43.5/40.3), the visual signal contributes little in the full-cue setting and the headline claim should be reframed. If the text-only model is substantially worse (e.g., more than 2 points BLEURT below full-cue), the concern is resolved and the visual contribution is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that adding automatically extracted contextual cues to visual signing features and pseudo-glosses significantly improves open-vocabulary SLT on BOBSL. The strongest reported result (Tab. 3) is the full-cue model Vid+PrevPred+PG+BG: BLEU-4 3.3, BLEURT 40.3, ROUGE-L 16.9, CIDEr 41.9, IoU 14.8, LLM 1.20. The ablation in Tab. 1 shows monotone gains when cues are added to video: Vid 41.0, +PG 41.8, +Prev 42.5, +BG 43.5 BLEURT on SENT-VAL. But every row contains video. There is no 'no-video, text-only' baseline — e.g., PrevPred+PG+BG without Vid, or PrevPred+BG without Vid. Since all cues except video are text strings, and since the LLM (Llama3-8B) is a strong English language model, the model could generate plausible English from the previous sentence and background keywords alone, with video contributing little. The analysis in Appendix A.3 reports only 3.4% IoU between background descriptions and ground-truth sentences, which the paper argues is not the whole signal; however, this low overlap also makes it harder to attribute gains to background. The concern is not that the model uses text cues — the paper's own claim is that context helps — but that the magnitude of the central contribution, 'visual signing plus context beats visual alone,' is not separable from 'text prior alone beats visual alone.' A text-only (no-video) baseline would settle whether the visual signal is necessary for the full-cue gain, and would clarify whether the improvement over video-only is attributable to the video-plus-text integration or simply to the strong text prior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based sign language translation (SLT) framework that combines visual signing features from a Video-Swin encoder with automatically extracted textual cues: pseudo-glosses, a background description from an image captioner, and the predicted translation of the previous sentence. All inputs are fed as a text-style prompt into a fine-tuned Llama3-8B decoder via LoRA and a learned mapping network. The authors evaluate on BOBSL and How2Sign, showing that each cue adds a positive contribution in ablations and that the full model outperforms GFSLT and Sign2GPT baselines on BOBSL, while achieving competitive results on How2Sign. They also introduce an LLM-based evaluation metric (GPT-4o-mini) that correlates strongly with human judgments.","tokens_in":27430,"tokens_out":3501,"duration_ms":39594,"significance":"If the central claim holds, the paper makes a strong empirical case that automatically extracted contextual cues, particularly previous-sentence translations and background scene descriptions, are valuable for open-vocabulary SLT from broadcast data. The work is also practically significant because it shows how to adapt a large pretrained LLM to sign language inputs without ground-truth glosses or spottings, and it provides a new evaluation metric that appears more aligned with human judgment than standard captioning metrics. The extensive ablations on BOBSL, the reproduction of prior baselines, and the code/project-page release are notable strengths. However, the missing no-video baseline and the backbone mismatch in the SOTA comparison prevent the current evidence from fully supporting the strong claims made in the paper.","major_comments":[{"comment":"The ablations always include the visual features (Vid), and there is no text-only baseline, such as PrevPred+PG+BG without Vid or PrevPred+BG without Vid. Since the previous sentence is a fluent English sentence, the pseudo-glosses are English words, and the background is an English keyword list, a strong LM like Llama3-8B could plausibly generate a reasonable English translation from the text cues alone, with the visual signal contributing little to the reported gains. The claim that 'context has additional relevant information beyond the signing video' is therefore not cleanly supported. Please add a no-video ablation on SENT-VAL (and ideally on SENT-TEST) to quantify the marginal contribution of the visual features to the full-cue model.","section":"§4.3, Table 1; Appendix B.2, Table A.5"},{"comment":"The comparison to GFSLT and Sign2GPT is confounded by the visual backbone: the proposed model uses a Video-Swin ISLR encoder pre-trained on BOBSL with a vocabulary of 8,697 signs, while the baselines are reproduced with their own visual encoders (I3D/DINOv2-based). The row 'Ours (Vid)' already surpasses the baselines, but this could be driven by the stronger Video-Swin features rather than by the proposed LLM-based translation architecture. To make the SOTA claim fair, please either train the baselines with the same Video-Swin features or report a version of the proposed model with an I3D backbone matching [3] and [67], and state the backbone configuration explicitly in the table.","section":"§4.4, Table 3"},{"comment":"The sliding window stride s is a free parameter that controls the number of visual features (F) and the amount of temporal detail, yet the value used in the main experiments is not reported. The text only says 'when s=2, we have on average 56 features,' without stating that s=2 is the chosen value for the BOBSL experiments, and no ablation over s is provided. Please specify the exact stride for BOBSL and How2Sign, and if possible include a sensitivity analysis, since changing s directly changes the computational cost and the information available to the LLM.","section":"§3.2, Visual features"}],"minor_comments":[{"comment":"The LLM-based evaluation metric is computed by GPT-4o-mini, and the translator itself is an LLM; although the correlation with human judgments (Fig. A.2) is reassuring, the metric may share systematic biases with LLM-generated translations. The paper could briefly note this potential bias and report the LLM scores with this caveat in mind.","section":"§4.1, LLM Evaluation"},{"comment":"The improvements from adding cues are reported without statistical significance tests. While +2.5 BLEURT and +0.27 LLM on a 1,973-sentence validation set are likely meaningful, providing error bars or a significance test (e.g., bootstrap) would strengthen the ablation claims.","section":"§4.3, Table 1"},{"comment":"The background descriptions overlap with ground-truth sentences by only 3.4% IoU, which the paper correctly interprets as not capturing the whole signal; however, the qualitative examples in Fig. 3 seem to show rather direct keyword matches (e.g., 'jellyfish', 'New Forest'). A quantitative analysis of how often background keywords appear in the final translations would help calibrate the reader's expectation.","section":"Appendix A.3"},{"comment":"The description of the Sincan [67] baseline is brief regarding the exact features and training setup; since the paper compares against it in the oracle setting, a sentence clarifying whether the same video features and decoder capacity are used would improve transparency.","section":"§4.2, Baselines"},{"comment":"The conclusion lists challenges such as error accumulation from previous-sentence predictions and noise in background descriptions; these are good, but the paper could also mention that the missing no-video baseline is a limitation of the current analysis, since it prevents isolating the visual contribution.","section":"§5, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a strong group and the experimental effort is substantial, including large-scale BOBSL training, baseline re-implementations, and a human-correlated evaluation metric. The main risk is that the central claim of 'context helps by going beyond visual signing' is not fully separated from the possibility that the LLM is mostly exploiting the strong English text priors. Adding a no-video ablation and a matched-backbone SOTA comparison would materially strengthen the paper. The current recommendation is major_revision rather than reject, because the missing controls appear feasible to add within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the central selling point — context beyond the signing video helps — is supported by the ablations, but the authors never run the one ablation that would tell you how much the video is actually contributing once the text cues are in the prompt. The full model (Vid+PrevPred+PG+BG) beats video-only by 2.5 BLEURT on SENT-VAL, but every row in Table 1 contains video. There is no text-only, no-video row, so you cannot separate \"video plus text integrates well\" from \"the LLM is just continuing the previous sentence and background keywords.\" Given Llama3-8B is a strong English language model, and the previous-sentence cue is a fluent English sentence, the text-prior story is a live alternative. The appendix's missing-cue table (A.6) still keeps video in every row, so it does not settle this.\n\nWhat's genuinely new: the specific combination of automatically extracted background descriptions, predicted previous-sentence translations, and pseudo-glosses as text inputs to a LoRA-tuned LLM for open-vocabulary SLT on BOBSL. That combination is not in Sincan et al., which used ground-truth previous subtitles and spottings. The ablations are extensive and the paper is honest about limitations (error accumulation, background noise, negation mistakes). The LLM-based metric is a useful addition: it correlates with human ratings (0.84), at least as well as BLEURT, though using GPT-4o-mini to grade an LLM-based translator is a mild circularity.\n\nSoft spots in proportion: (1) the missing text-only baseline is the main one; it would be cheap to add. (2) Table 3 compares their Video-Swin-based model against I3D/DINOv2 baselines, so part of the gain is backbone, not method. The paper reports Ours(Vid) separately, but it does not control for the backbone. (3) The absolute scores are low (BLEU-4 3.3 on BOBSL), so \"significant\" improvements mean significant within a hard task, not human-quality translation. (4) The background descriptions share only 3.4% IoU with ground truth; the paper argues the signal is semantic, which is plausible but not pinned down.\n\nBottom line: the context-helps claim is credible and the paper is worth engaging with. It deserves peer review, but the review should ask for a text-only baseline and, ideally, a backbone-matched comparison. I would bring it to a reading group for the LLM-as-translator discussion, and I would cite it for the contextual framework if I were working on SLT.","headline":"A solid, useful SLT paper that never runs the one ablation—text-only, no video—that would separate the LLM's text priors from the video's contribution; the context-helps claim is likely true but the margin is modest.","tokens_in":27833,"tokens_out":2495,"would_cite":true,"duration_ms":25638,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatically extracted context — background captions, previous-sentence translations, and pseudo-glosses — fed into a fine-tuned LLM substantially improves open-vocabulary sign language translation on BOBSL and transfers to How2Sign.","keywords":["sign language translation","contextual cues","large language model fine-tuning","open-vocabulary translation","BOBSL","British Sign Language","pseudo-glosses","How2Sign"],"falsifier":"Run the identical model with all text cues present but the visual tokens masked or replaced by random projections on BOBSL SENT-VAL; if BLEURT, IoU, and the LLM score do not drop materially below the full-cue model, the claim that the visual stream contributes beyond the text is refuted.","tokens_in":1605,"feed_emoji":"🤟","tokens_out":2189,"duration_ms":70249,"temperature":0.7,"pith_summary":"This paper tries to establish that sign language translation in the open-vocabulary setting improves when the translator is given the kind of context human interpreters use: what is happening in the background and what was just signed. It builds a single LLM-based decoder that receives visual signing features, automatically produced pseudo-glosses, keyword background descriptions, and the predicted translation of the previous sentence, all in one prompt. On the BOBSL British Sign Language benchmark, every cue adds a measurable gain and the full model outperforms previously reported results and two strong video-only baselines. The practical stake is that large interpreted TV archives, with their noisy subtitles, can support much better translation than video-only models achieve.","feed_headline":"Context cues lift sign translation to 3.3 BLEU-4 on BOBSL","feed_subtitle":"Background captions and previous sentences fed to a fine-tuned LLM beat video-only baselines on every metric.","key_machinery":"The load-bearing mechanism is a single prompt assembled for a pre-trained decoder-only LLM, Llama3-8B, fine-tuned with LoRA. Visual features from a Video-Swin isolated-sign recogniser are projected through a trainable 2-layer MLP into the LLM embedding space, while pseudo-glosses, background keyword lists, and the previous predicted sentence are inserted as text with task prompts. This machinery lets one language model fuse weak visual and textual evidence, using pretrained English fluency for grammar while the visual stream and the contextual cues jointly disambiguate referents, homonyms, tense, and signs that are omitted or underspecified in the video.","core_discovery":"On the paper's own terms, the central claim is that open-vocabulary sign language translation should be treated as a context-dependent discourse task rather than an isolated video-to-text problem. The proposed model feeds a fine-tuned pre-trained LLM a prompt containing visual signing features, pseudo-glosses from an isolated-sign recogniser, a keyword summary of captions describing the background footage, and the predicted translation of the previous sentence. On BOBSL SENT-TEST the full-cue model reports BLEU-4 3.3, BLEURT 40.3, ROUGE-L 16.9, CIDEr 41.9, IoU 14.8, and an LLM-based score of 1.20, surpassing state-of-the-art baselines and the video-only version of the same model on every metric, with the ablation study showing that each cue adds a positive contribution. The same framework, without background cues, also achieves competitive results on the How2Sign American Sign Language dataset.","pith_inferences":["A text-only variant with the visual tokens masked or removed is not reported, so the marginal contribution of the visual stream — as opposed to fluent English inferable from the previous sentence and background keywords — remains an open question.","The auto-regressive use of previous predictions may accumulate errors over long stretches of signing, and the paper's own experiment with two or three previous sentences shows only marginal gains.","Because background descriptions are extracted generically from the whole frame, they can introduce noise or even distract the model; a more targeted extraction, such as focusing on the region a signer points to, is a natural testable extension.","The recipe should transfer to other sign languages that have a large weakly-aligned subtitle corpus plus an isolated-sign recogniser, but not to sentence-level datasets lacking discourse or background context."],"forward_implications":["Open-vocabulary translation on broad-domain interpreted TV is feasible without manual glosses, since pseudo-glosses are used only as an input cue rather than supervision.","Background descriptions help resolve pronoun reference, place names, pointing gestures, and object names that are not signed or are ambiguous in the video.","Previous-sentence context helps with topic-comment structure, tense, and coherence, and using the model's own predictions at inference avoids needing ground-truth context.","The drop-cue augmentation makes the final model robust when some cues are missing at test time, so the system does not collapse when background or previous-sentence information is unavailable.","An LLM-based evaluation metric adapted from CLAIR correlates more strongly with human judgements than BLEU-4 on the paper's small human study, giving a more interpretable measure of translation quality."],"supporting_citations":[{"why":"Supplies the BOBSL dataset, its sentence-level evaluation splits, and the weak subtitle supervision the method trains on.","marker":"[3]"},{"why":"Provides the pre-trained Video-Swin ISLR model that yields both the visual features and the pseudo-glosses.","marker":"[62]"},{"why":"Supplies the automatic subtitle-signing alignment used to build the 689k training pairs from BOBSL.","marker":"[8]"},{"why":"Establishes the prior context-based SLT baseline with ground-truth previous sentence and spottings, against which the paper compares in the oracle setting.","marker":"[67]"},{"why":"Provides the Sign2GPT baseline and the LoRA fine-tuning precedent for adapting large pre-trained models to sign language translation.","marker":"[78]"},{"why":"Provides the GFSLT baseline, which is reproduced and adapted to BOBSL as a comparison method.","marker":"[87]"},{"why":"Supplies the Llama3-8B language model used as the fine-tuned decoder.","marker":"[25]"},{"why":"Supplies the BLIP2 image captioning model that generates the background descriptions.","marker":"[44]"},{"why":"Supplies the How2Sign dataset used to demonstrate generality beyond BSL TV broadcasts.","marker":"[23]"}],"fun_headline_variants":["Sign translation gets a context boost: BLEU-4 3.3 on BOBSL","Context is key: sign translation hits 3.3 BLEU-4 on BOBSL","Contextual cues beat video-only for sign translation, BLEU 3.3","Feeding LLM context improves sign translation, BLEU-4 3.3 on BOBSL","Sign translation improves with contextual cues: 3.3 BLEU-4 on BOBSL"],"cache_read_input_tokens":29824,"weakest_assumption_plain":"The central assumption is that the fine-tuned LLM genuinely combines the textual cues with the visual signing features, rather than generating plausible English mostly from the fluent previous-sentence text and background keywords, since no text-only baseline is reported.","fun_headline_variants_meta":{"raw":{"variants":["Sign translation gets a context boost: BLEU-4 3.3 on BOBSL","Context is key: sign translation hits 3.3 BLEU-4 on BOBSL","Contextual cues beat video-only for sign translation, BLEU 3.3","Feeding LLM context improves sign translation, BLEU-4 3.3 on BOBSL","Sign translation improves with contextual cues: 3.3 BLEU-4 on BOBSL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001248,"raw_usage":{"total_tokens":5132,"prompt_tokens":970,"completion_tokens":4162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":4037}},"tokens_in":586,"tokens_out":4162,"duration_ms":25892,"temperature":1.0,"reasoning_tokens":4037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:39:51.729873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical model with all text cues present but the visual tokens masked or replaced by random projections on BOBSL SENT-VAL; if BLEURT, IoU, and the LLM score do not drop materially below the full-cue model, the claim that the visual stream contributes beyond the text is refuted.","supporting_citations":[{"cited_title":"A tale of two languages: Large-vocabulary continuous sign language recognition from spoken language supervision.arXiv, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained Video-Swin ISLR model that yields both the visual features and the pseudo-glosses."},{"cited_title":"Is context all you need? scaling neural sign language translation to large domains of discourse","cited_arxiv_id":null,"evidence_quote":"Establishes the prior context-based SLT baseline with ground-truth previous sentence and spottings, against which the paper compares in the oracle setting."},{"cited_title":"Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation","cited_arxiv_id":null,"evidence_quote":"Provides the Sign2GPT baseline and the LoRA fine-tuning precedent for adapting large pre-trained models to sign language translation."},{"cited_title":"Gloss-free sign language translation: Improving from visual-language pretraining","cited_arxiv_id":null,"evidence_quote":"Provides the GFSLT baseline, which is reproduced and adapted to BOBSL as a comparison method."},{"cited_title":"The llama 3 herd of models","cited_arxiv_id":null,"evidence_quote":"Supplies the Llama3-8B language model used as the fine-tuned decoder."},{"cited_title":"BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the BLIP2 image captioning model that generates the background descriptions."},{"cited_title":"How2Sign: A large-scale multimodal dataset for continuous American Sign Language","cited_arxiv_id":null,"evidence_quote":"Supplies the How2Sign dataset used to demonstrate generality beyond BSL TV broadcasts."}],"review_version":1}