{"id":"9b5bfa29-a2a7-4d86-88f1-ca0532e258bb","arxiv_id":"2501.00935","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multiscale multi-head attention video transformer achieves 88.22% on NVGesture and 99.10% on Briareo for dynamic hand gesture recognition.","lead":"This paper presents a hand gesture recognition model that combines a video transformer with differently sized attention heads, so each head analyzes a different scale of the frame features. The authors report 88.22% accuracy on NVGesture and 99.10% on Briareo, beating several earlier methods, though not all published state-of-the-art scores on NVGesture.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The multiscale mechanism is confounded with a ~2x increase in attention width/parameters; no equal-capacity baseline is reported, so the causal claim is not yet established.","rationale":"The reader's weakest assumption is that the differently sized heads cannot be concatenated through a single W^O because Eq. 3 is not dimensionally updated. That is a real bookkeeping gap, but it is fixable: one can choose W^O with input dimension equal to the sum of the per-head output widths. The more scientifically load-bearing issue is that, once the dimensions are fixed, the MsMHA layer has about twice the attention width and parameter count of the standard transformer used as baseline. Thus any accuracy improvement could be due to capacity rather than to the pyramidal multiscale design. This directly threatens the causal claim in the abstract and conclusion. The proposed equal-width, equal-parameter ablation would isolate the multiscale effect. Because the concern is a missing control rather than demonstrated failure, the existing conditional verdict remains appropriate, so no verdict change is recommended.","tokens_in":7823,"tokens_out":8041,"duration_ms":78356,"concrete_test":"Reimplement the exact MsMHA layer with W^O of shape (Σ_{i=0}^{7} D/2^i) × D and run the NVGesture color-only protocol. Then run a controlled baseline: standard multi-head attention with the same total attention width W = Σ_{i=0}^{7} D/2^i split evenly across 8 heads and the same output projection W^O: W × D, fixing ResNet-18 features, optimizer, epochs, and seeds. Compare mean accuracy over at least 3 seeds. If the flat equal-width baseline matches or exceeds 81.42%, the multiscale-specific benefit is unsupported; if MsMHA clearly outperforms it, the concern is refuted. Also verify the released implementation's W^O shape and per-head value dimensions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-B and Table I define heads with Q/K/V widths D, D/2, ..., D/128, but Eq. 3 still uses the standard W^O ∈ R^{h·d_v × d_model}, whose input dimension is never reconciled with the sum Σ_{i=0}^{7} D/2^i ≈ 1.99D. Even if one fixes W^O to shape (≈1.99D) × D, the MsMHA attention layer has roughly twice the QKV and output-projection parameters of the baseline Transformer [12], which uses total width D. The reported gains in Table III — color 81.42% vs. 76.50%, optical flow 85.30% vs. 72.00% — therefore conflate the multiscale head design with added model capacity. No equal-parameter or single-scale control is reported, so the central claim that multiscale attention itself 'significantly improves' accuracy is not supported. Additionally, the state-of-the-art claim is contradicted by Table IV, where NAS1+NAS2 reaches 88.38% vs. the proposed 88.22% on NVGesture.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MsMHA-VTN, a video transformer for dynamic hand gesture recognition in which the per-head query/key/value dimensions are progressively halved across heads, forming a pyramid of multiscale attention. Frame-level features from a ResNet-18 backbone are processed through six transformer stages, and late fusion combines up to five input modalities (RGB, depth, infrared, surface normals, optical flow). The authors report an overall accuracy of 88.22% on NVGesture and 99.10% on Briareo, and compare with prior methods including the baseline Transformer of Eusanio et al.","tokens_in":8064,"tokens_out":5506,"duration_ms":44224,"significance":"The proposed multiscale-head idea is a reasonable and potentially useful modification of the standard transformer for gesture recognition, and the evaluation against two public benchmarks is a strength. If the architectural details are clarified and the gains are shown to be due to the multiscale mechanism rather than increased parameter count, the work would be a solid contribution. However, the current manuscript does not fully define the proposed layer, and the reported results are not statistically substantiated. The paper's own comparisons also do not support the claimed state-of-the-art status on NVGesture. With these fixes, the contribution could be of interest to the gesture recognition community.","major_comments":[{"comment":"The proposed MsMHA layer is not fully defined. Eq. (3) reuses the standard W^O ∈ R^{h·d_v × d_model}, but the concatenated output of the eight heads has dimension L × (D + D/2 + ... + D/128) ≈ L × 1.99D, not L × h·d_v. The manuscript never specifies the shape of W^O for MsMHA, nor how the query, key, and value tensors of differing widths are aligned in the attention computation. Without this specification, the architecture described in Section II-B cannot be reproduced, and the reported results cannot be clearly attributed to the described mechanism.","section":"Section II-B, Eqs. (3)-(5), Table I"},{"comment":"The accuracy improvements over the baseline Transformer [12] are confounded with a large increase in attention-layer parameters. Because the multiscale heads have widths D, D/2, ..., D/128, the total QKV projection width and the output projection width are approximately twice those of the standard MHA baseline. No equal-capacity or single-scale control is reported, so the claim that multiscale attention per se improves accuracy is not supported. The authors should include a baseline with the same total attention width but uniform head sizes, or otherwise control for capacity.","section":"Section III-B, Tables II-IV"},{"comment":"The claim of state-of-the-art performance is contradicted by the paper's own results. On NVGesture color-only input, NAS1 achieves 83.61% (Table III) versus 81.42% for MsMHA-VTN; on multi-modal NVGesture, NAS1+NAS2 achieves 88.38% (Table IV) versus 88.22% for MsMHA-VTN. The state-of-the-art claim should be removed or restricted to specific modalities and datasets (e.g., surface normals on NVGesture and the overall accuracy on Briareo).","section":"Abstract, Section III-B, Tables III-IV"},{"comment":"The statement that MsMHA-VTN 'significantly improves' performance is not supported by any statistical evidence. All reported accuracies are single-run point estimates without standard deviations, confidence intervals, or significance tests. The authors should either provide repeated-run statistics or replace 'significantly' with a more modest description of the observed differences.","section":"Conclusion, Section III-B"}],"minor_comments":[{"comment":"The row for the five-modality MsMHA-VTN result is typeset incorrectly: it reads 'color + depth + normals + 88.22%ir + flow' instead of listing the modalities and the accuracy separately.","section":"Table IV"},{"comment":"The caption contains a typo: 'Mutiscaled' should be 'Multiscaled'.","section":"Figure 2 caption"},{"comment":"The notation Q_{jM/2}, K_{jM/2}, V_{jM/2} is unclear; it should be written with explicit superscripts or subscripts (e.g., Q_j^{M/2}) and the meaning of M as 'dimension of the previous head tensor' should be defined more precisely.","section":"Section II-B, Eq. (5)"},{"comment":"The text states that the dimensions of the key and query vectors vary, but Table I lists the 'Attention Tensor size' for each head without specifying whether the value vectors also follow the same pyramid. Please clarify whether all three of Q, K, and V are scaled, or only Q and K.","section":"Section II-B"},{"comment":"The description of how the ResNet-18 backbone features are extracted (e.g., output dimension, temporal pooling, positional encoding added to the transformer input) is omitted; please provide these details or refer to the exact configuration in [12].","section":"Section III-A"},{"comment":"The paper does not report the number of parameters or FLOPs of the proposed model compared with the baseline, which would help assess the capacity confound raised in Major Comment 2.","section":"Section II-B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a promising architectural idea, but the current version has a load-bearing definition gap and unsupported performance claims. The authors need to clarify the MsMHA dimensions, add controlled capacity experiments, and temper the SOTA claim. The topic is within the scope of the journal, and the paper is likely salvageable with major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper has a small, concrete idea—give each attention head a different Q/K/V dimension, halving head by head, so the layer sees a pyramid of scales—and it reports a few points of improvement over the vanilla Transformer baseline on two public gesture datasets. The idea is not identical to MViT or CrossViT; those use pooling and cross-attention, whereas this is a per-head dimension pyramid. That is a legitimate variant, even if it is more of a parameter choice than a new principle.\n\nWhat it does well: the evaluation is on public benchmarks (NVGesture, Briareo), the comparison list is broad, and the late-fusion results are internally consistent. On Briareo, the proposed 99.10% is genuinely above the Transformer baseline (95.80% on normals, 97.20% on ir+normals). For a letter, the experimental setup is acceptable apart from what is missing.\n\nNow the soft spots, in decreasing order.\n\nFirst, the method as written does not close dimensionally. Eq. 3 still defines W^O as h*d_v x d_model, but Table I shows head dimensions D, D/2, ..., D/128. The concatenated width is about 1.99D, not 8*(D/8)=D, and the paper never says what W^O shape is or how the differently sized head tensors are concatenated. This is fixable by declaring W^O in R^{(sum_i d_i) x d_model}, but as printed the layer is underspecified.\n\nSecond, the headline comparison is confounded. If you implement the pyramid as described, the QKV projections and the output projection contain roughly twice the parameters of the baseline Transformer [12]. The paper's own tables show color 81.42 vs 76.50 and optical flow 85.30 vs 72.00, but no single-scale or equal-capacity control is run, so those gains could just be capacity, not multiscale attention per se. This is the strongest reason not to take the causal claim at face value.\n\nThird, the SOTA claim is overstated. On NVGesture, NAS1+NAS2 reaches 88.38%, above the paper's 88.22%, and the color-only result (81.42%) is below NAS1 (83.61%). The conclusion's 'significantly improve... compared to some existing methods' is technically true only against the vanilla transformer and a few others.\n\nThere are also no seeds or error bars, so the point accuracies are single-run numbers. That is minor for a letter but worth noting.\n\nWho should read this: someone tracking transformer variants for video, or a reviewer wanting a worked example of the capacity-confound critique. It deserves a serious referee—the dimensional issue and the confound are checkable, and the idea is coherent enough to warrant a revision. I would not cite it yet, but I would expect a competent referee to send it back for more work before accepting.\n\nRecommended call: engage, but require a corrected equation, a capacity-matched baseline, and at least a couple of seeds before the multiscale claim can stand.","headline":"Incremental transformer tweak with a plausible but under-specified multiscale head design; the gains versus the baseline are real but confounded by roughly twice the parameters, and the architecture has an unresolved dimension mismatch.","tokens_in":8615,"tokens_out":4043,"would_cite":false,"duration_ms":36524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video transformer with halved head dimensions reports 88.22% on NVGesture and 99.10% on Briareo.","keywords":["Dynamic hand gesture recognition","Video transformer","Multi-head attention","Multiscale attention","Pyramid attention","Multimodal fusion","NVGesture","Briareo"],"falsifier":"One concrete check: reimplement the model with the head sizes of Table I and verify that Concat(head1, ..., head8) $W^O$ is a defined matrix multiplication; if it is not, the mechanism as described cannot be what produced the reported accuracies. A second check: train the same pipeline on NVGesture with eight equal-dimension heads and the same total parameter budget; if the equal-head version matches or beats 88.22%, the improvement is not attributable to the multiscale pyramid.","tokens_in":7606,"feed_emoji":"✋","tokens_out":6941,"duration_ms":62359,"temperature":0.7,"pith_summary":"This letter tries to show that a simple change to the transformer's multi-head attention, letting each head attend at a different resolution by halving the query and key dimension head by head, improves dynamic hand gesture recognition. The proposed MsMHA-VTN builds a pyramid of attention scales, uses ResNet-18 frame features as in the baseline video transformer, and reports 88.22% overall accuracy on NVGesture and 99.10% on Briareo. The authors also show that fusing RGB, depth, infrared, surface normals, and optical flow at the decision level helps, and that surface normals alone are a strong single modality. If the claim is right, it means scale diversity inside the attention heads, rather than a larger or more complex network, is enough to capture the pose and size variations that make gestures hard to recognize.","feed_headline":"Attention pyramids hit 99.10% on hand-gesture video","feed_subtitle":"Halving head dimensions in a pyramid lifts gesture accuracy over the baseline transformer on two datasets.","key_machinery":"The central object is the Multiscaled Multi-Head Attention (MsMHA) module. In each of the six transformer stages, eight heads compute scaled dot-product attention as in Eq. (1), but instead of a common $d_k$, the query and key tensors are built with dimensions $L \\times D$, $L \\times D/2$, ..., $L \\times D/128$, forming a pyramid of decreasing resolutions. The head outputs are concatenated and projected through the output weight $W^O$ of Eq. (3), and the paper claims this lets each stage see gesture structure at multiple scales while keeping the later heads much smaller.","core_discovery":"The central discovery claimed by the paper is that a video transformer can learn gesture features at multiple scales by making the eight attention heads progressively smaller: head 1 works at $L \\times D$, head 2 at $L \\times D/2$, and so on down to $L \\times D/128$, instead of giving every head the same dimension. Six stacked stages of this Multiscaled Multi-Head Attention (MsMHA) extract frame-level features from a ResNet-18 backbone, and the authors report that this design beats the original transformer on every modality they test and yields overall accuracies of 88.22% on NVGesture and 99.10% on Briareo. The paper interprets these results as evidence that pyramidal, multiscale attention handles the variation in hand pose, size, and shape that makes dynamic gesture recognition hard.","pith_inferences":["Beyond the paper's datasets, the same pyramid-head design could be tested on other video classification tasks with scale variation, such as sign language recognition or action recognition; the paper only evaluates hand gestures.","Because the later heads are much smaller, MsMHA likely uses fewer parameters and less compute per stage than a standard eight-head transformer, though the paper does not report parameter counts or FLOPs; a direct efficiency comparison would be needed to confirm.","The strong normals-only results suggest a practical single-sensor deployment path: if depth-based normals alone nearly match multimodal fusion on Briareo, one can drop RGB and IR streams in settings where a depth camera is available.","An ablation that keeps total parameters fixed while varying head dimensions would separate the multiscale effect from capacity effects; the paper does not include such an ablation."],"forward_implications":["The standard transformer baseline [12] is improved on every single modality tested: color, depth, optical flow, normals, and infrared all score higher with the pyramid attention.","Surface normals computed from depth are a particularly informative input: 86.21% on NVGesture and 98.67% on Briareo with normals alone, already close to the best multimodal scores.","Late fusion helps on NVGesture, where all five input streams give the best result (88.22%), but on Briareo three streams (color, infrared, normals) are enough, and adding more modalities does not improve accuracy.","The reported accuracies compare favorably with published methods on both datasets, including recurrent 3D CNNs, multi-stream CNNs, and other transformer or neural-architecture-search based systems."],"supporting_citations":[{"why":"Defines scaled dot-product and multi-head attention, the base module that MsMHA modifies.","marker":"[5]"},{"why":"Provides the baseline video transformer for dynamic hand gesture recognition, including the ResNet-18 feature extractor and training setup.","marker":"[12]"},{"why":"Supplies the NVGesture dataset and the R3D-CNN baselines that appear throughout the comparison tables.","marker":"[4]"},{"why":"Introduces the Briareo dataset and the C3D-HG and LSTM-HG baselines used for comparison.","marker":"[29]"},{"why":"Provides NAS1 and NAS2 neural-architecture-search results that the proposed model is compared against on NVGesture.","marker":"[14]"},{"why":"Provides the MTUT multimodal training baselines that the proposed model aims to beat on NVGesture.","marker":"[25]"}],"fun_headline_variants":["Pyramidal attention heads push gesture video accuracy to 99.10%","Multiscale attention transformer hits 99.1% on hand gestures","Shrinking attention heads per stage beat baseline on NVGesture and Briareo","Six-stage video transformer with shrinking heads scores 99.10% on Briareo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the eight differently sized attention heads can be concatenated and multiplied by a single output projection $W^O$, exactly as written in Eq. (3); the paper does not specify the size of that projection, and if the shapes do not match, the proposed transformer layer is not well defined.","fun_headline_variants_meta":{"raw":{"variants":["Pyramidal attention heads push gesture video accuracy to 99.10%","Multiscale attention transformer hits 99.1% on hand gestures","Shrinking attention heads per stage beat baseline on NVGesture and Briareo","Six-stage video transformer with shrinking heads scores 99.10% on Briareo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2781,"prompt_tokens":848,"completion_tokens":1933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1848}},"tokens_in":464,"tokens_out":1933,"duration_ms":12772,"temperature":1.0,"reasoning_tokens":1848,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:40:32.760101+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check: reimplement the model with the head sizes of Table I and verify that Concat(head1, ..., head8) $W^O$ is a defined matrix multiplication; if it is not, the mechanism as described cannot be what produced the reported accuracies. A second check: train the same pipeline on NVGesture with eight equal-dimension heads and the same total parameter budget; if the equal-head version matches or beats 88.22%, the improvement is not attributable to the multiscale pyramid.","supporting_citations":[{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Defines scaled dot-product and multi-head attention, the base module that MsMHA modifies."},{"cited_title":"A Transformer-based network for dynamic hand gesture recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline video transformer for dynamic hand gesture recognition, including the ResNet-18 feature extractor and training setup."},{"cited_title":"Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"Supplies the NVGesture dataset and the R3D-CNN baselines that appear throughout the comparison tables."},{"cited_title":"Hand gestures for the human-car interaction: The Briareo dataset,","cited_arxiv_id":null,"evidence_quote":"Introduces the Briareo dataset and the C3D-HG and LSTM-HG baselines used for comparison."},{"cited_title":"Searching multi-rate and multi-modal temporal enhanced networks for gesture recognition,","cited_arxiv_id":null,"evidence_quote":"Provides NAS1 and NAS2 neural-architecture-search results that the proposed model is compared against on NVGesture."},{"cited_title":"Improving the perfor- mance of unimodal dynamic hand-gesture recognition with multimodal training,","cited_arxiv_id":null,"evidence_quote":"Provides the MTUT multimodal training baselines that the proposed model aims to beat on NVGesture."}],"review_version":1}