{"id":"b9228c41-e031-4899-9c2d-914d32a9d068","arxiv_id":"2412.13609","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sign-IDD generates sign language poses by disentangling joint coordinates into bone direction and length attributes inside a gloss-conditioned diffusion model, reporting state-of-the-art scores on PHOENIX14T and USTC-CSL.","lead":"This paper introduces Sign-IDD, a diffusion model that turns sign language glosses into 3D body poses by representing each bone as a direction plus a length, rather than only joint coordinates. It reports improved accuracy and semantic scores on two sign language benchmarks, PHOENIX14T and USTC-CSL.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label ground truth on PHOENIX14T is the load-bearing weakness: pose metrics and BLEU/WER are computed against, and trained on, the same OpenPose/Zelinka outputs, so reported gains could reflect fitting label errors rather than better signing.","rationale":"I considered missing error bars and single-seed ablations as an alternative concern, but pseudo-label validity is logically upstream: even a many-seed rerun cannot correct a biased yardstick. The reader's weakest_assumption identifies the same load-bearing issue, and I agree with it. The paper does not include a formal limitations paragraph, but its own Figure 4 caption is an in-scope admission that the ground-truth poses contain inaccurate labels due to motion blur, which reinforces the concern. The central claim requires that the reported improvements in pose accuracy and semantic faithfulness are genuine; on PHOENIX14T, every reported number is defined relative to pseudo-labels produced by the same OpenPose-plus-lifting pipeline that generated the training targets. If those pseudo-labels carry systematic errors, the ID and ACD modules may simply be better at imitating those errors, and the claimed causal role of the modules is not independently established. USTC-CSL and the internal ablations keep the approach plausible, so I do not recommend changing the reader's CONDITIONAL verdict; I recommend keeping it until the PHOENIX14T evaluation is validated against independent 3D labels.","tokens_in":13485,"tokens_out":6574,"duration_ms":61664,"concrete_test":"Select a subset of PHOENIX14T test videos and obtain independent 3D pose labels, either through manual annotation or a different 3D pose estimator (e.g., VideoPose3D or FrankMocap). Recompute MPJPE, MPJAE and FID for Sign-IDD, Base, GEN-OBT and at least one diffusion baseline against these independent labels, and also run a back-translator trained only on independent or clean 3D poses to compute BLEU/WER. If Sign-IDD's advantage over baselines persists under independent labels, the pseudo-label concern is resolved; if its margin shrinks or reverses, the reported PHOENIX14T gains are likely an artifact of matching flawed pseudo-labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"On PHOENIX14T, the Implementation Details section states that 3D target poses are produced by OpenPose 2D keypoints lifted with the Zelinka and Kanis (2020) skeletal correction model, because PHOENIX14T has no pose labels. This choice is upstream of nearly every quantitative claim: the same pseudo-labels are used (i) as training targets in Ljoint (Eq. 15) and Lbone (Eq. 16), (ii) as the reference for MPJPE, MPJAE and FID, and (iii) as the signal used to train or evaluate the NSLT back-translator that yields BLEU and WER. Systematic errors in pose extraction, especially in hand articulation and limb orientation, are therefore baked into both the objective being optimized and the measuring stick. The paper itself concedes inaccuracies in Figure 4's caption, labeling instances with 'inaccurate labels' and 'motion blur'. Since the ID and ACD modules specifically target joint associations and bone orientation, their apparent gains on PHOENIX14T could reflect fitting the pseudo-label skeleton's bias rather than producing semantically correct signs. This weakens the central attribution claim on the main benchmark, especially because all ablations in Tables 3 and 4 are also on PHOENIX14T. USTC-CSL provides partial, cleaner support, but the headline claim about pose accuracy and semantic fidelity rests substantially on PHOENIX14T results that are not independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sign-IDD, a gloss-to-pose diffusion model for sign language production. Instead of regressing only 3D joint coordinates, it converts poses into a '4D bone representation' of unit direction vectors and bone lengths (Eq. 4), concatenates this 4D representation with the 3D pose, and feeds the result through an attribute separation/control module inside a diffusion denoiser. Training uses a joint loss (Eq. 15) and a bone-orientation loss (Eq. 16). The method is evaluated on PHOENIX14T and USTC-CSL with BLEU/ROUGE/WER via NSLT back-translation plus FID, MPJPE, and MPJAE, and the paper reports improvements over several published baselines and over its own ablations.","tokens_in":13795,"tokens_out":6888,"duration_ms":69137,"significance":"If the reported results are reliable, the paper makes a modest but useful contribution: it shows that an explicit relative/bone representation can improve pose generation in SLP, and it provides a clean, parameter-light modification of a diffusion backbone. The code is released, the paper includes a ground-truth row to calibrate the back-translation metrics, and the ablations separately test both the ID and ACD components. The main uncertainties are the quality of the pseudo-label ground truth on PHOENIX14T and the absence of variance estimates; these currently prevent me from judging whether the claimed gains are due to the proposed mechanism or to fitting label artifacts.","major_comments":[{"comment":"The PHOENIX14T pose ground truth is produced by OpenPose 2D keypoints lifted with the Zelinka-Kanis skeletal correction model, because the dataset has no native pose labels. These pseudo-labels serve simultaneously as the training targets for Ljoint and Lbone and as the reference for MPJPE, MPJAE, FID, and the NSLT-based BLEU/WER scores. Any systematic bias in the pseudo-labels, especially in hand articulation and limb orientation, is therefore baked into both the objective and the measuring stick. The paper itself notes that the ground truth contains 'inaccurate labels' and 'motion blur' in Figure 4. To support the central attribution claim, please add ablations and comparisons on a dataset with native pose annotations (or otherwise verify against independent annotations), report a quantitative sensitivity analysis using an independent pose estimator on a subset of PHOENIX14T, or provide a human evaluation of pose correctness.","section":"Implementation Details; Eqs. (15)-(16); Table 1"},{"comment":"All ablations are single-run and are reported only on PHOENIX14T. The increments attributed to the modules are small in absolute terms (e.g., Base+ID vs Base+ID+ACD on TEST BLEU-1: 23.46 vs 24.80, and WER 77.53 vs 76.66), so without multiple seeds, error bars, or significance tests the claim that ID and ACD are each responsible for the gains is not statistically supported. Please report run-to-run variance or significance tests, and, if possible, repeat the ablations on USTC-CSL.","section":"Tables 3 and 4"},{"comment":"The comparison is incomplete in a way that affects the headline claim. NAT-ATM, NAT-EAM, and G2P-DDM have no FID/MPJPE/MPJAE entries, DET* has no WER/FID/MPJPE/MPJAE, and GCDM lacks FID/MPJPE/MPJAE on both DEV and TEST. The abstract and conclusions claim that the method 'outperforms SOTA methods,' but this is only partially supported by the metrics that are actually reported. Please either obtain and report these metrics with a shared protocol (including the same NSLT model and pose representation) or explicitly restrict the claim to the metrics where all methods are compared.","section":"Table 1"}],"minor_comments":[{"comment":"The fused representation in Eq. (6) is written as [pt, p′t] ∈ R7, but pt has J joints while p′t has B = J − 1 bones according to Eq. (4). Please state explicitly that a zero 4D root bone is included so that the two sequences have the same length; otherwise the concatenation is undefined.","section":"Eq. (6) and Reverse Process"},{"comment":"The description of the inference timestep is confusing: 'Initiated at T, the timestep for each iteration is computed as T = T − (1−i/I)' reuses the symbol T. Please clarify the schedule and the role of i.","section":"Eq. (12) and inference procedure"},{"comment":"The term 'iconicity' is used in a nonstandard way. In sign linguistics, iconicity refers to a resemblance between form and meaning, not to invariance of semantics under a coordinate transformation. Please either define the term clearly in the paper or use a more neutral name such as 'bone disentanglement.'","section":"Introduction and Related Work"},{"comment":"There is a typo in the parameter row of Table 4: 'angi = 5' should be 'i = 5'.","section":"Table 4"},{"comment":"Figure 1 labels both the traditional 3D representation and the proposed 4D representation as '(b)', which is confusing; the second panel should be '(c)'.","section":"Figure 1"},{"comment":"The paper states that several typical solutions were tested 'under a fair setting' on USTC-CSL, but does not describe this setting. Please provide implementation details for the reimplemented baselines (e.g., training protocol, pose representation, and any hyperparameter tuning).","section":"USTC-CSL comparison"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-scoped and the core idea is simple and testable, but the evaluation is currently the main weakness. The PHOENIX14T pseudo-label issue is load-bearing because the same labels are used for training, direct pose metrics, and back-translation metrics. The missing baseline metrics and single-run ablations add to the uncertainty. I would be willing to reconsider after the authors strengthen the evaluation with independent pose verification, variance estimates, and a more complete comparison table."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chao, quick read of Sign-IDD. The one thing to know: the method is a sensible incremental step in gloss-to-pose diffusion, but the main benchmark is weaker than the tables make it look. PHOENIX14T has no native 3D pose labels, so the authors train and evaluate on OpenPose 2D keypoints lifted to 3D with Zelinka-Kanis. The same pseudo-labels are used as training targets, as the reference for MPJPE/MPJAE/FID, and as the signal feeding the NSLT back-translator that produces BLEU/WER. The paper even admits in Figure 4 that some ground-truth labels are inaccurate due to motion blur. That means the reported gains on the main benchmark could partly be fitting the biases of the pseudo-label skeleton rather than producing better signing.\n\nWhat's actually new: the ID module converts 3D joints to a 4D bone representation (normalized direction plus length), and the ACD module separates and re-controls those attributes inside a diffusion model conditioned on gloss embeddings. The bone-direction/length decomposition is not new on its own — it's been used in skeleton-based action recognition — but the specific integration in a diffusion G2P model with bone supervision is a reasonable contribution. The ablations are clear: each module improves BLEU-1 and WER over the base, and the qualitative figures show crisper hand details. Code is released. That's real work.\n\nThe soft spots beyond the pseudo-label issue: no error bars or repeated runs, so we don't know if the margins are stable. Several SOTA baselines in Table 1 are missing metrics (e.g., G2P-DDM has no FID/MPJPE), which makes the head-to-head comparison less sharp. The ablation is only on PHOENIX14T, so the clean USTC-CSL dataset doesn't test the modules. And the best lambda=0.1 sits between 0.01 and 1 with a sharp drop at 1; a sensitivity curve around 0.1 would be more convincing.\n\nThe stress-test note holds up. It's the load-bearing weakness, and it's not fixed by the paper's own acknowledgments. That said, the paper is a fair, honest piece of incremental research. It belongs in the SLP subfield conversation, and the USTC-CSL results provide some independent support. I'd send it to review, but require the authors to address the pseudo-label problem — for example, evaluate on a manually verified subset, report pose metric variance, and include the missing baseline numbers. A serious referee should engage.","headline":"Solid, incremental sign-language pose generation with a real evaluation weakness: the PHOENIX14T ground-truth poses are pseudo-labels, and the paper's headline claims rest on them.","tokens_in":14336,"tokens_out":2203,"would_cite":false,"duration_ms":20168,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Representing sign poses as bone directions and lengths, instead of raw joint coordinates, makes a diffusion model produce more accurate and semantically faithful sign poses.","keywords":["sign language production","gloss-to-pose","diffusion models","3D pose generation","bone representation","pose disentanglement","skeleton constraints","PHOENIX14T"],"falsifier":"Generate sign poses on a dataset with true 3D labels (or motion capture) using both Sign-IDD and a 3D-coordinate-only diffusion baseline and compare MPJPE/MPJAE; if the bone-based model does not still win on clean ground truth, the reported advantage is an artifact of pseudo-label noise. A simpler version: train with USTC-CSL real poses and check whether the improvement over baseline matches the PHOENIX14T improvement.","tokens_in":13270,"feed_emoji":"🤟","tokens_out":7653,"duration_ms":61050,"temperature":0.7,"pith_summary":"Sign Language Production from glosses usually treats a pose as a set of independent 3D joint coordinates and fits those coordinates directly. This paper argues that such a representation ignores the relative spatial relationships between joints, which are what make signs readable. It proposes Sign-IDD, a diffusion framework that first disentangles each 3D pose into bones, each described by a 3D directional vector and a 1D length, and then uses those bone attributes to control pose generation alongside the original coordinates. The claim, supported by experiments on PHOENIX14T and USTC-CSL, is that adding this bone-level constraint yields poses that are semantically closer to the intended signs and show fewer finger overlaps and limb-orientation errors. If correct, this makes text-to-sign production more reliable without requiring new data.","feed_headline":"Bone directions and lengths improve AI-generated sign language","feed_subtitle":"A diffusion model that treats each limb as direction plus length lifts BLEU and lowers WER on two sign benchmarks.","key_machinery":"The workhorse is the 4D bone representation obtained by iconicity disentanglement. For each parent-child pair of joints along the body topology, the representation is $q^* = (q_c - q_p)/||q_c - q_p||_2$ for the 3D unit direction and $m = ||q_c - q_p||_2$ for the length, with the neck as root joint; this turns absolute joint coordinates into relative bone descriptors. The ACD module is the second piece: it concatenates the 3D and 4D pose streams, injects gloss embeddings through multi-head attention, then uses an attribute separation layer to split features back into coordinate and attribute streams and an attribute control layer to let bone attributes refine the coordinate features during denoising. A bone constraint loss supervises the predicted directions directly. Together these components give the model an explicit handle on bone orientation and length, which is what carries the argument.","core_discovery":"The central claim is that converting a 3D joint pose into a 4D bone representation, where every bone carries the normalized direction from parent to child joint and the Euclidean distance between them, exposes the skeletal structure that coordinate regression hides. On PHOENIX14T TEST, Sign-IDD reports BLEU-1 24.80, WER 76.66, FID 2.46, MPJPE 47.19, and on USTC-CSL it reports BLEU-1 65.26 and WER 50.15, beating the compared baselines on most metrics. The paper attributes these gains to two components: the Iconicity Disentanglement (ID) module, which derives the 4D bone representation and feeds it together with the 3D coordinates into the denoiser, and the Attribute Controllable Diffusion (ACD) module, which separates bone direction and length features from coordinate features and uses them as control signals during reverse diffusion. It also adds a bone-orientation loss $L_{bone}$ to the usual joint loss. The paper's conclusion is that modeling both joint positions and their relative associations is what improves pose accuracy and semantic consistency.","pith_inferences":["Because PHOENIX14T has no real pose labels, the paper's MPJPE/MPJAE numbers on that corpus measure error against pseudo-ground-truth produced by OpenPose and a lifting model; a fair test of absolute pose accuracy would require clean 3D labels, for example motion capture or manual annotation.","The same direction-plus-length disentanglement could transfer to other articulated motion generation tasks, such as hand gesture synthesis or human motion prediction, where relative joint geometry carries much of the meaning.","A control experiment that gives a 3D-coordinate-only baseline the same extra parameters would show whether the gains come from the disentangled representation itself or merely from added network capacity; the paper does not isolate this.","The USTC-CSL results, which use real pose labels, are the cleaner evidence for the method's pose-level claims; if future work focuses there, the bone-representation advantage can be verified without pseudo-label confounds."],"forward_implications":["Sign pose generators can be made more precise by explicitly modeling bone orientation and length, not only joint coordinates; the reported gains on BLEU, WER, FID, MPJPE, and MPJAE point to fewer semantic errors and fewer distorted hands and limbs.","The 4D bone representation is a cheap side channel that works with existing gloss-to-pose diffusion architectures, so other SLP systems could adopt it without changing their overall training regime.","The ablation results show that both the ID and ACD modules contribute beyond the diffusion baseline, and that the bone constraint weight $\\lambda = 0.1$ matters: too small or too large a weight degrades performance, so there is a practical tuning range for skeletal supervision.","The method is tested on a German sign corpus and a Chinese sign corpus, suggesting the bone-disentanglement benefit is not language-specific."],"supporting_citations":[{"why":"Supplies OpenPose 2D joint extraction used to create PHOENIX14T pseudo-labels.","marker":"(Cao et al. 2017)"},{"why":"Lifts 2D joints to 3D poses that serve as PHOENIX14T ground truth.","marker":"(Zelinka and Kanis 2020)"},{"why":"Provides the PHOENIX14T dataset and the NSLT model used to back-translate poses for BLEU and WER evaluation.","marker":"(Camgoz et al. 2018)"},{"why":"Source of the joint loss and non-autoregressive baselines NAT-ATM and NAT-EAM.","marker":"(Huang et al. 2021)"},{"why":"G2P-DDM diffusion baseline that Sign-IDD compares against and improves on BLEU metrics.","marker":"(Xie et al. 2024)"},{"why":"Provides the gloss-driven conditional diffusion baseline and the gloss encoder used in Sign-IDD's condition integration.","marker":"(Tang et al. 2024)"},{"why":"D3DP-sign diffusion-based baseline with multi-hypothesis aggregation that Sign-IDD compares against.","marker":"(Shan et al. 2023)"},{"why":"Pose-estimation work showing bone length and orientation constraints help, the inspiration for the bone representation.","marker":"(Cai et al. 2024)"},{"why":"Autoregressive progressive transformer G2P baseline whose coordinate-only representation is the target of the critique.","marker":"(Saunders, Camgoz, and Bowden 2020b)"}],"fun_headline_variants":["4D bone encoding sharpens AI sign-language output","From joints to bones: a new angle on sign-language AI","Bone direction and length: the missing ingredient in sign video generation","Sign language AI learns from limb directions and lengths","Bone-aware diffusion model beats joint-only sign pose generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All PHOENIX14T results rest on computer-estimated 3D poses standing in for true sign poses; if those pseudo-labels are systematically biased, the reported pose-accuracy improvements are not measurements of real pose accuracy.","fun_headline_variants_meta":{"raw":{"variants":["4D bone encoding sharpens AI sign-language output","From joints to bones: a new angle on sign-language AI","Bone direction and length: the missing ingredient in sign video generation","Sign language AI learns from limb directions and lengths","Bone-aware diffusion model beats joint-only sign pose generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4221,"prompt_tokens":1068,"completion_tokens":3153,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":3072}},"tokens_in":684,"tokens_out":3153,"duration_ms":22202,"temperature":1.0,"reasoning_tokens":3072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:57:15.809033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate sign poses on a dataset with true 3D labels (or motion capture) using both Sign-IDD and a 3D-coordinate-only diffusion baseline and compare MPJPE/MPJAE; if the bone-based model does not still win on clean ground truth, the reported advantage is an artifact of pseudo-label noise. A simpler version: train with USTC-CSL real poses and check whether the improvement over baseline matches the PHOENIX14T improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies OpenPose 2D joint extraction used to create PHOENIX14T pseudo-labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lifts 2D joints to 3D poses that serve as PHOENIX14T ground truth."},{"cited_title":"C.; Hadfield, S.; Koller, O.; Ney, H.; and Bowden, R","cited_arxiv_id":null,"evidence_quote":"Provides the PHOENIX14T dataset and the NSLT model used to back-translate poses for BLEU and WER evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the gloss-driven conditional diffusion baseline and the gloss encoder used in Sign-IDD's condition integration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"D3DP-sign diffusion-based baseline with multi-hypothesis aggregation that Sign-IDD compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pose-estimation work showing bone length and orientation constraints help, the inspiration for the bone representation."}],"review_version":1}