{"id":"1758c29a-ad17-418b-8a8c-9ae7135d7beb","arxiv_id":"2411.12901","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A compact transformer with a modified Conformer-style convolution and APE/CoPE position encodings reports near-SOTA gloss-free sign language translation on PHOENIX14T with 0.57 to 6.44 million parameters.","lead":"Signformer, a 3.88-million-parameter transformer, reportedly matches a 7-billion-parameter model on German sign language translation using 1800 times fewer parameters. The paper argues that efficient, from-scratch architectures can replace pretrained LLMs for gloss-free sign language AI.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 23.43 is a test-selected maximum from an unreported configuration sweep; the 0.03-BLEU margin over SignLLM is within run-to-run noise, so the 2nd-place and 1807x claims are not yet established.","rationale":"The reader's verdict was REJECT, and the weakest assumption identified was that the reported numbers are not typical performance because the paper only presents configurations that improve performance. My stress-test sharpens this into a concrete statistical flaw: the 23.43 BLEU-4 for Signformer-Full is the best value selected across a hidden-size/CoPE sweep, with the authors explicitly saying they omit configurations that hurt performance. The comparison with SignLLM hinges on a 0.03 BLEU difference, which is almost certainly smaller than seed-to-seed variation on PHOENIX14T. Additionally, the reported Test BLEU (23.43) exceeds Dev BLEU (21.92) by 1.51 points for the flagship configuration, an unusual pattern that strongly suggests test-set peeking during configuration selection. A dev-locked re-run with multiple seeds would settle whether the reported result is reproducible or an artifact. This does not change the reader's REJECT verdict; it strengthens the central reason for it. I do not raise concerns about author intent, and I credit the paper for clearly documenting its ablation protocol, even though the stated protocol is precisely what undermines the leaderboard claim.","tokens_in":11792,"tokens_out":11428,"duration_ms":115967,"concrete_test":"Re-run the Table 4 sweep using only the PHOENIX14T dev set for model selection: train each hidden-size/CoPE configuration (at least 128/256, with/without CoPE), pick the best by dev BLEU, lock it, and report test BLEU over 5 seeds. If the locked configuration is not 256/non-CoPE, or if the mean test BLEU of the locked config is not >= 23.40, the reported 23.43 is a test-selection artifact and the 2nd-place claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a 0.03-BLEU win over SignLLM (23.43 vs 23.40) on PHOENIX14T. That margin is smaller than typical run-to-run variance, yet the paper gives no seeds, error bars, or model-selection protocol. Section 4.1 states 'we chose to feature configurations offering tangible benefits only,' and Table 4's note says 'We neglect configurations worsening performance as that's trivial to present.' This is an admission that the headline 23.43 is the maximum of a configuration sweep (hidden size x CoPE) evaluated on the test set, not a pre-registered architecture. Supporting anomaly: the reported Full configuration (256 hidden, no CoPE) has Test BLEU 23.43 vs Dev BLEU 21.92; a 1.51-point test-over-dev gap is atypical and consistent with selecting on test labels. Without error bars or a held-out selection procedure, the '2nd place' and '1807x fewer parameters' claims are not established; the decisive margin over SignLLM is within noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Signformer, a from-scratch transformer architecture for gloss-free sign language translation, combining a convolutional module, gloss attention, and contextual position encoding (CoPE). It reports that Signformer-Full (3.88M parameters) achieves BLEU-4 23.43 on the PHOENIX14T test set, which it claims is a new 2nd place on the gloss-free leaderboard, approximately matching SignLLM (7B parameters, BLEU-4 23.40) with 1807x fewer parameters. The paper also presents linguistic arguments about German word order to justify the choice of absolute position encoding, and reports a lighter 0.57M-parameter configuration (Signformer-Feather) that outperforms several existing methods.","tokens_in":12034,"tokens_out":4438,"duration_ms":43220,"significance":"If the reported results were robustly established, the paper would make a valuable contribution to efficient sign language translation by demonstrating that a compact from-scratch model can approach the performance of billion-parameter pretrained systems. The emphasis on edge-AI deployment and the analysis of linguistic structure are worthwhile research directions. However, the current evidence is not convincing: the central numbers appear to be selected from an unreported configuration sweep, no error bars or multiple-seed results are provided, the optimizer is changed across configurations, and no code or project link is given. The manuscript therefore does not currently justify its headline claims.","major_comments":[{"comment":"The paper explicitly states, \"We neglect configurations worsening performance as that's trivial to present,\" and in §4.1 that \"we chose to feature configurations offering tangible benefits only.\" This indicates that the reported Test BLEU-4 of 23.43 for Signformer-Full is the maximum of an unreported configuration sweep, not the performance of a single pre-specified model. No error bars, standard deviations, or multiple-seed results are given, so the 0.03-BLEU margin over SignLLM (23.43 vs. 23.40) is within typical run-to-run noise. Moreover, the Dev BLEU-4 of 21.92 versus Test BLEU-4 of 23.43 for this configuration shows a 1.51-point test-over-dev gap, which is atypical and consistent with selecting the configuration on the test set. The central claims of \"2nd place\" and \"1807x smaller\" are therefore not established.","section":"§4.1 and Table 4"},{"comment":"The optimizer is not held fixed: §4.1 states that SophiaG was used for Mid and Full configurations with hidden sizes above 128, while AdamW was adopted for the Feather models with hidden size 64, and that optimizers were chosen to \"acquire the best configuration results.\" This confounds all ablation comparisons: the effects attributed to CoPE, hidden size, or the convolution module could instead be due to the optimizer change. Since no seeds or variance estimates are reported, none of the ablation tables provide a controlled comparison that supports the paper's conclusions about the architecture components.","section":"§4.1 and Tables 1–4"},{"comment":"The position-encoding ablation is not a clean test of the linguistic hypothesis. Table 2's experiments for RPE and CoPE were run with the Multihead Attention module because \"Gloss Attention doesn't support RPE,\" while the final architecture uses Gloss Attention. Thus the comparison conflates the attention mechanism with the position encoding, and the claimed superiority of APE may be an artifact of this mismatch. Additionally, the linguistic rationale (German's strict word order favoring APE) is presented as a post hoc explanation after the empirical results are known; the paper provides no pre-registered or predictive test of this hypothesis, so the \"nature analysis\" does not independently support the design choice.","section":"§3.3 and Table 2"},{"comment":"The headline comparison against SignLLM rests on a 0.03-BLEU difference (23.43 vs. 23.40), which is smaller than typical run-to-run variance for neural translation models, and the paper provides no confidence intervals or repeated runs. Furthermore, the parameter counts for competing methods are partly estimates or obtained through personal communication, as the table note indicates, and the methods are compared across different training protocols and resource budgets. Without variance estimates and a clearly defined model-selection procedure, the claims of approximate equivalence to SignLLM and of a 1807x parameter reduction are not statistically supported.","section":"§5.1 and Table 5"}],"minor_comments":[{"comment":"The manuscript contains numerous typos and grammatical errors, e.g., \"sophiscated\" (Abstract), \"metricuously\" (Contributions), \"paradign\" (§2.3), \"applicaiton\" (§2.1), \"combinded\" (§3.2), \"syntatic\" (§4.2), \"leadeboard\" (§2.3), and \"ture\" (§4). A thorough proofreading pass is needed.","section":"Throughout"},{"comment":"The table has inconsistent naming: the method is called \"STMC-T\" in §2.2 but listed as \"SMTC-T\" in Table 5; the table note also references \"SMTC-T\" while the text says \"STMC-T.\" The relationship between the confirmed 46.4M parameter count for STMC and the reported 192* for the ensembled STMC-T should be clarified.","section":"§5.1, Table 5"},{"comment":"The phrase \"We release official code and project page available here\" is not accompanied by an actual link or repository identifier anywhere in the manuscript. For a reproducibility-focused claim, a working URL or a note explaining the lack of release is necessary.","section":"Abstract and §1"},{"comment":"The paper states that CSL-Daily was not included due to access restrictions, which is an understandable limitation, but the claim that \"we have ensured both datasets are representative and consistent\" is unsupported because no experiments on CSL-Daily are presented. The generalizability claim should be toned down or substantiated with results on at least one additional dataset.","section":"§4 and §5"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim is the problem. The paper says Signformer-Full (3.88M params) matches SignLLM (7B) at 23.43 vs 23.40 BLEU-4 on PHOENIX14T, a 0.03-point gap that is pure noise. Worse, the authors admit in Table 4 that they 'neglect configurations worsening performance' and in Section 4.1 that they feature configurations offering 'tangible benefits only.' So the headline number is a maximum over a hidden-size × CoPE sweep, evaluated on the test set, with no error bars or seeds. The test-over-dev gap (23.43 test vs 21.92 dev) is atypically large and smells like selection on test labels. Without code, seeds, or a fixed model-selection protocol, the '2nd place' and '1807x' claims collapse.\n\nWhat's genuinely new and good: the architecture is a clean combination of GASLT's deformable attention, a modified Conformer conv block, and APE+CoPE. The conv modification (Table 1) gives a real ~1 BLEU boost at the same parameter count, which is a small but legitimate contribution. The from-scratch stance is refreshing in a field leaning on LLMs, and a 0.57M-parameter model that breaks 20 BLEU is interesting if it holds up. The ablations, while selective, do cover the main design choices.\n\nThe soft spots beyond the cherry-picking: the optimizer switches between AdamW (Feather) and SophiaG (larger configs), so the hidden-size scalability comparisons are not controlled. Parameter counts for competitors are partly from emails and estimates, which is acceptable but adds uncertainty. The linguistic analysis of word order is post hoc rationalization—APE 'fits' German syntax, but the only evidence is the ablation table, and the CoPE story is similarly retrofitted. None of this is fatal to the core idea, but it means the empirical case is unproven.\n\nWho should read this: researchers in efficient vision-language or edge AI, and anyone studying test-set overfitting in leaderboard culture. The paper deserves peer review, because the efficiency hypothesis is worth testing properly. But as submitted, the evidence is not reproducible, and the central claim is a selection artifact. A serious referee should send it back for major revision: release code, report multiple seeds and error bars, and specify a single pre-registered configuration before test evaluation.","headline":"The paper's headline result—23.43 BLEU with 3.88M parameters, '2nd place' and '1807x smaller' than SignLLM—is a test-selected maximum from an unreported configuration sweep, so the central claim is not established, though the from-scratch efficiency angle is worth a serious look.","tokens_in":12502,"tokens_out":2695,"would_cite":false,"duration_ms":28999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Signformer, a from-scratch transformer with 3.88 million parameters, reports BLEU-4 23.43 on the PHOENIX14T gloss-free sign language translation benchmark, matching the 7-billion-parameter SignLLM at 23.40.","keywords":["sign language translation","gloss-free translation","from-scratch transformer","parameter efficiency","position encoding","video-to-text","PHOENIX14T","edge AI"],"falsifier":"Recompute BLEU-4 for Signformer-Full and SignLLM with one shared evaluation script (same tokenization, beam size, and detokenization); if the 0.03-point gap reverses or widens materially, the claimed parity and 2nd-place rank collapse. A simpler check is to rerun Signformer-Full several times and see whether 23.43 is the typical score rather than the best of a few runs, since the paper reports only configurations that improve performance.","tokens_in":11606,"feed_emoji":"🤟","tokens_out":9785,"duration_ms":87233,"temperature":0.7,"pith_summary":"Signformer seeks to prove that gloss-free sign language translation does not need billion-parameter pretrained language models. The paper introduces a compact transformer trained from scratch on the PHOENIX14T German corpus and reports that its 3.88-million-parameter Full configuration reaches BLEU-4 23.43, slightly above the 7-billion-parameter SignLLM at 23.40 while being about 1800 times smaller. Its 0.57-million-parameter Feather configuration still scores BLEU-4 20.02, above most existing gloss-free methods. If these numbers hold, the result makes deployable, low-cost sign-to-text translation plausible and reframes the field's recent turn to LLM backbones and extra pretraining as one possible route rather than the necessary one.","feed_headline":"A 3.88M-parameter model matches a 7B sign translator","feed_subtitle":"Signformer reports BLEU-4 23.43 on PHOENIX14T, matching SignLLM's 23.40 with 1807x fewer parameters.","key_machinery":"The load-bearing mechanism is the Signformer encoder-decoder stack: a reworked 1D convolution module (pointwise-depthwise-pointwise with a single ReLU6, inside a symmetric layer-normalization flow) that extracts local temporal gesture features; GASLT-style gloss attention that encodes gloss-like units in latent space without explicit gloss labels; absolute position encoding, chosen because German word order is rigid; and contextual position encoding (CoPE) applied in cross-attention to make positions sensitive to local, non-manual cues such as facial expressions. The claim is that this combination, trained with raw spatial and word embeddings, produces the reported parameter-efficiency and BLEU-4 gains.","core_discovery":"The paper's central claim is that a from-scratch, gloss-free sign language translation architecture can reach near the top of the 2024 gloss-free leaderboard without any pretrained model, extra dataset, gloss annotation, or NLP transfer trick. Signformer-Full reports BLEU-4 23.43 on PHOENIX14T with 3.88M parameters, placing second and edging out SignLLM's 23.40 with 7012M parameters; Signformer-Feather reports BLEU-4 20.02 with 0.57M parameters. The authors attribute this to a direct video-to-text transformer whose convolutional module, gloss attention, and dual position-encoding scheme are designed from an analysis of sign-language structure, and they treat the result as establishing an Edge-AI baseline for sign language translation.","pith_inferences":["A direct corollary the paper does not spell out: if a 3.88M-parameter decoder matches a 7B LLM backbone, the LLM's contribution to this benchmark may be largely replaceable; feeding SignLLM's visual tokens into a small decoder would quantify how much of its BLEU comes from the backbone.","The paper's linguistic argument makes a cross-lingual prediction: on a sign language with flexible word order, such as Japanese Sign Language, CoPE should contribute relatively more and APE relatively less; running the same architecture on a second corpus when available would test this.","The reported non-monotonicity of CoPE (helpful below hidden size 128, harmful above) suggests a testable general hypothesis about contextual position encodings in video-to-text tasks, not just a tuning detail for this model."],"forward_implications":["Gloss-free sign language translation can be tackled as an architecture problem rather than a scaling problem; LLM backbones and CLIP-style pretraining are not required for competitive BLEU.","A model under 4 million parameters can sit at or near the top of the gloss-free leaderboard, making CPU-only or mobile deployment realistic.","Language-aware choice of position encoding matters: on a rigid-word-order target like German, absolute position encoding outperforms relative and contextual encodings alone.","Adding CoPE helps most in small hidden-size configurations and hurts above size 128, so the efficiency gain is specific to compact models.","Signformer-Feather at 0.57M parameters gives a concrete lightweight baseline that future small-model research can compare against."],"supporting_citations":[{"why":"Supplies the SL-Transformer baseline, joint training protocol, and preprocessed PHOENIX14T data on which Signformer is built.","marker":"[5]"},{"why":"Provides the GASLT gloss attention mechanism and the reproduced vanilla gloss-free baseline that Signformer extends.","marker":"[41]"},{"why":"Defines SignLLM, the 7B-parameter LLM-backed system whose BLEU-4 23.40 and 7012M parameter count are the main comparison.","marker":"[14]"},{"why":"Introduces CoPE, the contextual position encoding Signformer applies in cross-attention.","marker":"[13]"},{"why":"Supplies the Conformer convolution module that Signformer reinvents for gesture feature extraction.","marker":"[16]"},{"why":"Provides the PHOENIX14T German dataset used for all reported BLEU and ROUGE scores.","marker":"[12]"},{"why":"Reports the Sign2GPT pseudo-gloss pretraining approach that Signformer claims to surpass for second place.","marker":"[40]"}],"fun_headline_variants":["Signformer: from-scratch edge AI for sign translation","0.57M-parameter sign model rivals 7B LLM","Signformer matches 7B model with 1807x fewer params","Edge AI sign translator beats LLM-based methods","From-scratch sign transformer achieves leaderboard 2nd"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the parameter counts and BLEU scores for every competing method, some obtained by email from other authors and some estimated, are measured on the same terms as Signformer's; if they are not, the '2nd place' and '1807x smaller' claims lose their basis.","fun_headline_variants_meta":{"raw":{"variants":["Signformer: from-scratch edge AI for sign translation","0.57M-parameter sign model rivals 7B LLM","Signformer matches 7B model with 1807x fewer params","Edge AI sign translator beats LLM-based methods","From-scratch sign transformer achieves leaderboard 2nd"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1729,"prompt_tokens":990,"completion_tokens":739,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":652}},"tokens_in":606,"tokens_out":739,"duration_ms":6897,"temperature":1.0,"reasoning_tokens":652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:03:04.479011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute BLEU-4 for Signformer-Full and SignLLM with one shared evaluation script (same tokenization, beam size, and detokenization); if the 0.03-point gap reverses or widens materially, the claimed parity and 2nd-place rank collapse. A simpler check is to rerun Signformer-Full several times and see whether 23.43 is the typical score rather than the best of a few runs, since the paper reports only configurations that improve performance.","supporting_citations":[{"cited_title":"Sign language transformers: Joint end-to- end sign language recognition and translation, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the SL-Transformer baseline, joint training protocol, and preprocessed PHOENIX14T data on which Signformer is built."},{"cited_title":"Gloss attention for gloss-free sign language translation, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the GASLT gloss attention mechanism and the reproduced vanilla gloss-free baseline that Signformer extends."},{"cited_title":"Llms are good sign language translators, 2024","cited_arxiv_id":null,"evidence_quote":"Defines SignLLM, the 7B-parameter LLM-backed system whose BLEU-4 23.40 and 7012M parameter count are the main comparison."},{"cited_title":"Contextual position encoding: Learning to count what’s important, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces CoPE, the contextual position encoding Signformer applies in cross-attention."},{"cited_title":"Conformer: Convolution-augmented transformer for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the Conformer convolution module that Signformer reinvents for gesture feature extraction."},{"cited_title":"Piater, and Hermann Ney","cited_arxiv_id":null,"evidence_quote":"Provides the PHOENIX14T German dataset used for all reported BLEU and ROUGE scores."},{"cited_title":"Sign2gpt: Leveraging large language models for gloss-free sign language translation, 2024","cited_arxiv_id":null,"evidence_quote":"Reports the Sign2GPT pseudo-gloss pretraining approach that Signformer claims to surpass for second place."}],"review_version":1}