{"id":"aab1c44e-1736-4b21-a9af-aee554c2e6a7","arxiv_id":"2505.02304","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GSP-MC combines LLM-generated multipart sign descriptions with multi-positive contrastive learning to improve skeleton-based sign language recognition, reporting 97.1% on SLR-500 and 97.07% on AUTSL.","lead":"Researchers built a sign language recognition system that uses AI-generated written descriptions of each sign as extra training signals, then aligns those descriptions with the body's joint movements to improve accuracy. It reports top accuracy on Chinese and Turkish sign language benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic-supervision claim is untested: because positives are defined by label alone, the contrastive loss could act as label-driven clustering even if CLIP embeddings for Chinese/Turkish descriptions are semantically uninformative.","rationale":"The reader's weakest assumption concerned the frozen CLIP text encoder's cross-lingual quality. I agree that this is a live risk, but the deeper issue is structural: Eq. (9) defines positives by label membership alone, so even a semantically vacuous text encoder would supply a valid clustering target. The contrastive loss is therefore a supervised label-regularizer in the worst case, and the paper does not isolate whether the LLM description text itself carries the benefit. The ablation hierarchy in Tables 4 and 5 is evidence that the pipeline components matter, especially the knowledge base and optimized prompts, so the paper is not without support. However, without a non-semantic control, the central claim that generative LLM descriptions provide effective auxiliary supervision is not fully established. The verdict should remain CONDITIONAL, with the condition expanded to include this control experiment and a multilingual-encoder comparison, in addition to the code/data release and error-bar reporting the reader requested.","tokens_in":15243,"tokens_out":4277,"duration_ms":57576,"concrete_test":"On SLR-500, run the same GSP-MC pipeline with descriptions replaced by label-conditioned non-semantic strings, e.g., 'class <label>' or random token sequences paired to labels, encoded by the same frozen CLIP encoder. If accuracy stays within roughly 0.5% of the joint-only 95.81% or the 4-stream 97.1%, the semantic content of LLM descriptions is not load-bearing. Additionally, rerun AUTSL and SLR-500 with a multilingual text encoder such as multilingual CLIP; if accuracy drops substantially, the reader's cross-lingual concern is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that LLM-generated, expert-grounded descriptions provide effective auxiliary supervision. The mechanism in Section 3.3.1 defines the positive distribution p_i, Eq. (9), purely by the indicator I_match(s,t_i), which is 1 exactly when the skeleton and text share a sign label. The text encoder is frozen (Section 3.2.3), so the only trainable pathway is the skeleton encoder. Consequently, L_con in Eq. (10) can improve classification even if the CLIP text features carry no semantic content: it pulls all skeleton features of a class toward an arbitrary fixed text cluster and separates classes, which is essentially supervised contrastive learning with label-determined positives. The paper's ablation in Table 4 shows that raw LLM output without a knowledge base degrades accuracy, so text content can matter; but no control removes semantic content while preserving label pairing. The reader's cross-lingual concern is one concrete instance: CLIP is pre-trained on English image-text pairs, and the paper provides no evidence that Chinese and Turkish sign descriptions yield discriminative embeddings. If the text features are uninformative, the state-of-the-art numbers may be due to the label-structured contrastive loss rather than to generative LLM descriptions, undercutting the paper's stated contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces GSP-MC, a method for isolated sign language recognition from skeleton data. It proposes a retrieval-augmented generation (RAG) pipeline that uses expert-validated sign language dictionaries and an LLM to produce global, synonym, and body-part descriptions for each sign, then embeds these descriptions with a frozen CLIP text encoder. A skeleton encoder (CTR-GCN with 87 HR-Net keypoints) is trained jointly with a cross-entropy classification loss and a multi-positive contrastive loss (Eqs. 9-12) that aligns each skeleton feature with all text descriptions sharing the same sign label. Experiments on SLR-500 and AUTSL report state-of-the-art accuracies of 97.1% and 97.07%, respectively, together with ablations over description source, prompt optimization, multipart contrastive learning, and keypoint selection.","tokens_in":15415,"tokens_out":5146,"duration_ms":60836,"significance":"If the empirical results are reproducible, the paper demonstrates a practical way to inject expert-grounded linguistic descriptions as auxiliary supervision for SLR while keeping the inference-time model as a single skeleton encoder. The work is among the first to use generative LLMs for isolated sign language recognition, and the ablation chain in Tables 4-6 provides useful evidence that the knowledge base and part-level alignment contribute to accuracy. The cross-lingual evaluation on Chinese and Turkish is also a strength. However, the central claim that the gains come from the semantic content of the generated descriptions is not yet fully established, the headline 4-stream fusion result is outside the ablation analysis, and the keypoint-selection comparison is not internally controlled. These issues are addressable and do not by themselves invalidate the approach.","major_comments":[{"comment":"The positive indicator I_match(s,t_i) is defined solely by whether the skeleton and text share the same sign label, and the text encoder is frozen (Section 3.2.3). As a result, L_con in Eq. (10) is mathematically compatible with a supervised contrastive loss over label-defined positives even if the CLIP text embeddings carry no usable semantic content for Chinese or Turkish descriptions. Table 4 shows that replacing KB-grounded texts with raw LLM output degrades accuracy, which suggests content matters, but this is not a decisive control because the hallucinated texts are still label-paired target vectors. To support the paper's stated contribution, please add a control that removes semantic content while preserving label pairing (e.g., random fixed text embeddings assigned per label, or label-consistent shuffled descriptions), and ideally compare the frozen CLIP text encoder with a multilingual text encoder for Chinese and Turkish. This is the load-bearing evidence for the semantic-supervision claim.","section":"Section 3.3.1, Eq. (9)"},{"comment":"The headline SLR-500 result of 97.1% is obtained with a 4-stream fusion (joint/joint_motion/bone/bone_motion), but the ablation study in Table 5 is reported only on joint data, and no controlled ablation shows the marginal contribution of GSP-MC at the 4-stream level. The text in Section 4.2.2 attributes the AUTSL improvement of 1.36% to the multi-positive contrastive mechanism, but no ablation table supports that number for the fused configuration. Please report a controlled ablation at the exact configuration used for the headline numbers, including the fusion baseline without GSP-MC.","section":"Section 4.2.1, Table 2 versus Section 4.3.2, Table 5"},{"comment":"The keypoint-selection analysis compares the proposed 87-keypoint configuration with results from external methods (MASA, CoSign, etc.) that use different backbones, training protocols, and supervision signals. The 'Parts + Multipart' column therefore does not isolate the effect of the keypoint count. The 'all 133' row (59.22%) is especially difficult to interpret without knowing whether it uses the same encoder, preprocessing, and training procedure. Please provide an internal ablation in which the number of keypoints is varied within the same GSP-MC training pipeline.","section":"Section 4.3.3, Table 6"},{"comment":"The mathematical presentation of the contrastive objective is too imprecise for reproduction. In Eq. (7), the subscript a is defined as a = sum_{b=1}^{i-1} m_b + j, but the expression q^{s->t}_{ia} is written with an index j in the numerator; Eq. (8), q^{t->s}(t) = q^{s->t}(t)^T, is not a well-defined operation on a distribution; Eq. (9) mixes a per-sample p_i with a matrix in R^{B x M}; and the claim that the global-description case has M = B despite also using synonym variants T_s is not reconciled with the three-term sum in Eq. (11). Please rewrite this section with consistent indices, dimensions, and normalization.","section":"Section 3.3.1, Eqs. (7)-(10)"}],"minor_comments":[{"comment":"The claim of being the 'first work to integrate generative LLMs into SLR tasks' is overstated, since reference [32] (Sign2GPT) already integrates an LLM for gloss-free sign language translation. Please qualify the novelty claim to isolated, skeleton-based SLR or to the specific recognition setting addressed here.","section":"Abstract and Introduction"},{"comment":"The text says the raw LLM output causes a 0.27% accuracy degradation, while Table 4 shows 93.57% versus 93.85%, which is a 0.28% drop. Please make the numbers consistent.","section":"Section 4.3.1, Table 4"},{"comment":"Figure 6 contains a long string of '/uni...' tokens that appears to be corrupted or unrendered text. If this is intended to present class names or per-class statistics, it should be replaced with readable labels and a proper caption.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper makes strong state-of-the-art claims but does not include a code release or a reproducibility appendix. Given that the central semantic-supervision claim depends on the quality of the generated descriptions, I would encourage the editor to ask the authors for the generated description sets and for the controls described in the major comments, as these would substantially strengthen the paper's evidential value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competent, incremental contribution to skeleton-based sign language recognition. Its new piece is pairing retrieval-augmented generation with expert-verified sign language dictionaries to produce multipart descriptions, then using a multi-positive contrastive loss to align skeleton part features to those descriptions. The multi-positive formulation is a clean generalization of CLIP-style single-positive contrastive learning, and the ablations show each component—RAG, optimized prompts, synonym variants, part-level descriptions—adds a modest but consistent gain. The improvements over prior SOTA are small (about 0.2–0.8 points on SLR-500 and AUTSL), which is typical for this crowded benchmark space.\n\nWhere it earns credit: the authors actually tested the failure mode of raw LLM generation, showing it hurts accuracy unless anchored to a knowledge base; that is a useful negative result. The attention visualizations are plausible and align with the described mechanism. The writing is clear and the method is reproducible in principle, though no code or data is released. The citation pattern is unremarkable; the few self-citations are to their own prior encoder work, not a problem.\n\nSoft spots, in rough order of importance. First, the headline 97.1% uses a 4-stream fusion (joint, joint_motion, bone, bone_motion), but every ablation is single-stream joint only. The reader cannot tell how much of the fused gain comes from the contrastive text supervision versus the standard multi-stream ensembling trick. Second, no code, no data release, and no error bars; for gains of a few tenths of a point, that limits confidence in the exact numbers. Third, the cross-lingual assumption is real: a frozen CLIP text encoder pre-trained mostly on English image-text pairs is used for Chinese and Turkish descriptions with no control or comparison to a multilingual encoder. Finally, Table 6's keypoint comparison mixes external methods' configurations with different backbones, so the +1.40% 'multipart' column is not an apples-to-apples comparison.\n\nOn the stress-test concern: yes, the positives in Eq. (9) are defined by label alone, so the contrastive loss would still cluster same-class samples even if the text embeddings carried no semantic content. But the Table 4 ablation shows raw LLM output without the knowledge base degrades accuracy, which means text content matters beyond mere label pairing. That does not fully isolate the semantic mechanism—a control with random per-class text would be ideal—but it is enough to keep the central claim plausible.\n\nThe paper deserves a serious peer review. It is not groundbreaking, but it is honest, well-scoped engineering that advances a practical subfield. The main requests should be code/data release and an ablation of the 4-stream fusion.","headline":"Solid incremental contribution; the multi-positive contrastive formulation is clean, but the headline fused-number ablation and missing artifacts keep confidence conditional.","tokens_in":16016,"tokens_out":3525,"would_cite":true,"duration_ms":41061,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-written sign descriptions push sign recognition to 97.1 percent","keywords":["Sign Language Recognition","Contrastive Learning","Generative Large Language Model","Modality Fusion","Retrieval-Augmented Generation","Skeleton-Based Recognition","Multipart Description"],"falsifier":"Replace the frozen CLIP text encoder with a multilingual text encoder, or translate the Chinese and Turkish descriptions into English before encoding, and rerun the same GSP-MC pipeline on SLR-500 and AUTSL. If accuracy does not drop, the cross-lingual success is not due to the original text encoder's multilingual reliability; if the gains vanish, that assumption is confirmed as load-bearing.","tokens_in":14947,"feed_emoji":"🤟","tokens_out":5437,"duration_ms":60184,"temperature":0.7,"pith_summary":"This paper claims that sign language recognition can be improved by having large language models write multipart descriptions of what each sign looks like, then training a skeleton-based recognizer to match those texts. The proposed method, GSP-MC, anchors the language model's output in expert-validated sign dictionaries through retrieval-augmented generation, producing global, synonym, and part-level descriptions for each sign. A dual-encoder architecture aligns skeleton features with all descriptions of the same sign as multiple positives, using a KL-divergence contrastive loss that generalizes single-positive contrastive learning. On the Chinese SLR-500 dataset the method reaches 97.1% accuracy and on the Turkish AUTSL dataset 97.07%, both state-of-the-art results. If correct, this means generative text descriptions can serve as cheap, scalable auxiliary supervision for sign language recognition without adding inference-time cost.","feed_headline":"LLM-written sign descriptions push sign recognition to 97.1%","feed_subtitle":"Multi-positive contrastive learning aligns skeleton motion with expert-grounded text in Chinese and Turkish, at no extra inference cost.","key_machinery":"The load-bearing mechanism is a multi-positive contrastive loss defined over probabilistic matches. For each skeleton feature $s_i$, the true correspondence distribution is $p_i = I_{\\text{match}}(s_i,t_i)/\\sum_c I_{\\text{match}}(s_i,t_c)$, where $I_{\\text{match}}$ is 1 when text and skeleton share a label; the model minimizes a symmetric KL divergence between this distribution and the softmax of cosine similarities in both directions, $s\\to t$ and $t\\to s$. This generalizes standard single-positive contrastive learning and is applied hierarchically: global skeleton features align with global and synonym texts, while five part-pooled skeleton features align with part-specific texts. The texts themselves come from a generative prompt pipeline that uses retrieval-augmented generation over expert sign dictionaries to suppress hallucination and decompose each sign into part-level descriptions.","core_discovery":"The central claim is that a sign's meaning can be transferred into a skeleton-based recognizer through text: the authors generate, for every sign label, a global description, synonym variants, and part-specific descriptions covering left hand, right hand, face, mouth, and body, then train a graph-convolution skeleton encoder so that its global and part-pooled features are pulled toward all text embeddings of the same sign and pushed away from others. The text encoder is frozen, and the contrastive objective treats every description sharing a label as a matched positive via a probabilistic assignment distribution, optimized with a symmetric KL divergence. The paper reports state-of-the-art accuracy of 97.1% on Chinese SLR-500 and 97.07% on Turkish AUTSL, with ablations showing that raw LLM descriptions without grounding hurt accuracy, while retrieval-augmented grounding, optimized prompts, synonym variants, and multipart contrastive learning each add improvement. At inference only the skeleton encoder runs, so the gains come without extra computational overhead.","pith_inferences":["The frozen text encoder is the untested linchpin: since it was pretrained mostly on English image-text data, a multilingual text encoder or English-translated descriptions could plausibly match or beat the reported gains, a comparison the paper does not run.","The class-wise pattern implies a ceiling set by description quality: signs whose generated texts are vague or wrong lose accuracy, so investing in better prompting or human verification for low-gain classes should transfer directly into accuracy gains.","The same multi-positive, part-aware contrastive formulation could apply to other many-to-many vision-language problems, such as medical images with multiple partial reports or action recognition with multiple valid captions, wherever one visual sample has several legitimate text descriptions.","A natural next test is continuous sign language: if part-level text alignment helps isolated signs, it may also help segment or annotate continuous signing, which the authors list as future work."],"forward_implications":["Because the text encoder and part branches are discarded at inference, the model keeps the speed of a plain skeleton recognizer while gaining text-supervised semantics.","Signs with concrete, decomposable descriptions gain most (the paper reports +17.33% for 'ice cream' and +14.67% for 'experience'), so recognition quality becomes tied to description informativeness.","The same pipeline works for Chinese and Turkish, suggesting that expert-grounded generative descriptions are a language-agnostic supervision signal rather than a per-language annotation effort.","Ablations show that ungrounded LLM output hurts accuracy by 0.28%, and grounding in a knowledge base recovers 1.04% beyond the visual baseline, so the method's success depends on retrieval grounding rather than on the LLM alone.","Multi-positive contrastive learning contributes around 1.36% on AUTSL by capturing inter-sign variation, consistent with the claim that many-to-many text-skeleton alignment beats one-positive alignment."],"supporting_citations":[{"why":"Supplies the earlier LLM-description approach for skeleton action recognition that this work adapts to sign language with retrieval-augmented grounding.","marker":"[33]"},{"why":"Provides the frozen text encoder that maps Chinese and Turkish descriptions into the embedding space aligned with skeleton features.","marker":"[28]"},{"why":"Provides the graph-convolution backbone with channel-wise topology refinement used to encode skeleton sequences.","marker":"[3]"},{"why":"Defines the single-positive contrastive objective that the proposed KL-based multi-positive loss generalizes.","marker":"[27]"},{"why":"Supplies the Chinese SLR-500 benchmark and its signer-independent split used for the 97.1% result.","marker":"[11]"},{"why":"Supplies the Turkish AUTSL benchmark and its split used for the 97.07% result.","marker":"[29]"},{"why":"Supplies the HR-Net pose estimator that yields the 87 keypoints grouped into five body parts.","marker":"[30]"},{"why":"One of the authoritative Chinese sign dictionaries used as retrieval knowledge to ground generated descriptions.","marker":"[6]"},{"why":"One of the authoritative Chinese sign dictionaries used as retrieval knowledge to ground generated descriptions.","marker":"[7]"},{"why":"The Turkish National Sign Language Dictionary used as retrieval knowledge for the Turkish descriptions.","marker":"[18]"}],"fun_headline_variants":["First SLR with LLM text: 97.1% accuracy without extra inference","LLM descriptions boost sign recognition to 97.1% on two datasets","Text-guided skeleton learning hits 97.1% sign accuracy, no overhead","Multi-positive contrastive learning: sign recognition reaches 97.1%","Retrieval-grounded LLM prompts drive sign recognition to new high"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the frozen text encoder produces reliable semantic embeddings for Chinese and Turkish sign descriptions even though it was pretrained mostly on English image-text data; if those embeddings are poor, the contrastive alignment cannot carry the reported cross-lingual gains.","fun_headline_variants_meta":{"raw":{"variants":["First SLR with LLM text: 97.1% accuracy without extra inference","LLM descriptions boost sign recognition to 97.1% on two datasets","Text-guided skeleton learning hits 97.1% sign accuracy, no overhead","Multi-positive contrastive learning: sign recognition reaches 97.1%","Retrieval-grounded LLM prompts drive sign recognition to new high"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001364,"raw_usage":{"total_tokens":5544,"prompt_tokens":966,"completion_tokens":4578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":4477}},"tokens_in":582,"tokens_out":4578,"duration_ms":38221,"temperature":1.0,"reasoning_tokens":4477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:55:38.127595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the frozen CLIP text encoder with a multilingual text encoder, or translate the Chinese and Turkish descriptions into English before encoding, and rerun the same GSP-MC pipeline on SLR-500 and AUTSL. If accuracy does not drop, the cross-lingual success is not due to the original text encoder's multilingual reliability; if the gains vanish, that assumption is confirmed as load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the earlier LLM-description approach for skeleton action recognition that this work adapts to sign language with retrieval-augmented grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the frozen text encoder that maps Chinese and Turkish descriptions into the embedding space aligned with skeleton features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Turkish AUTSL benchmark and its split used for the 97.07% result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the authoritative Chinese sign dictionaries used as retrieval knowledge to ground generated descriptions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the authoritative Chinese sign dictionaries used as retrieval knowledge to ground generated descriptions."}],"review_version":1}