REVIEW 4 major objections 5 minor 39 references
HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read HyLiFormer claims that moving skeleton features into a Poincaré ball lets a transformer run attention in linear time while keeping accuracy close to quadratic-attention models.
desk verdict Well-motivated but broken at the one equation that matters: Eq. 13 is asserted, not derived, so the claimed linear hyperbolic attention is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the HLA reformulation in Eqs. (12)–(13): $Q$, $K$, $V$ are taken as shifted slices of the same input, and the output is $V_i = Q_i \cdot \mathrm{Sim}(K_j^T V_j)$, with $\mathrm{Sim}$ chosen as an exponential kernel. This reordered computation aggregates key–value pairs into a fixed-size vector in feature space, avoiding the $O(N^2)$ similarity matrix. The companion HTC embedding, $x^{\mathcal{B}} = -\frac{1}{\kappa}\tanh(-\kappa\|x\|)\cdot \frac{x}{\|x\|}$, is what places the data in the Poincaré ball and encodes hierarchy through the tanh compression of magnitudes. Lemmas 4.1 and 4.2 are the arguments that the output of each module respects the Poincaré ball constraint, which is what allows the model to claim it is genuinely operating in hyperbolic space.
What would settle it
The decisive check is an ablation that replaces the similarity kernel $\mathrm{Sim}$ in Eq. (13) with a fixed scalar (or replaces $Q,K,V$ with identical shifted copies) and measures accuracy on NTU RGB+D 120 X-Sub; if accuracy does not drop, the HLA module is not computing attention and the claimed linear-attention mechanism is not what carries the performance.
Extended reading notes
Core claim
The paper's central claim is that the Poincaré model can host a linear attention mechanism that is both well-defined and accurate for skeleton data. The HTC module converts each skeleton point via $\mathbf{x}^{\mathcal{B}} = -\frac{1}{\kappa}\tanh(-\kappa\|\mathbf{x}\|)\cdot \frac{\mathbf{x}}{\|\mathbf{x}\|}$, and the HLA module then sets $Q=\mathbf{x}[1:]$, $K=\mathbf{x}[2:]$, $V=\mathbf{x}[3:]$ and computes $V_i = Q_i \cdot \mathrm{Sim}(K_j^T V_j)$ with a softmax/exponential kernel, so the $N{\times}N$ similarity matrix is never materialized. Lemmas 4.1 and 4.2 assert that both transformations keep points inside the Poincaré ball, and the complexity statement is reduced from $O(N^2F)$ to $O(NF^2)$. The experiments on NTU RGB+D and NTU RGB+D 120 are offered as evidence that this efficiency comes with only a small accuracy cost relative to quadratic-attention transformers.
Load-bearing premise
Everything rests on Eq. (13) being a genuine attention mechanism with the same modeling power as softmax attention; if that premise is false, the HLA module is a temporal shift or gating operation and the central contribution collapses.
Editorial extensions
If this is right
- Self-attention cost drops from $O(N^2F)$ to $O(NF^2)$; for fixed feature dimension $F$ the cost is linear in sequence length, which is the paper's headline efficiency result.
- Because HLA avoids the $N{\times}N$ attention matrix, longer skeleton sequences can be processed with a fixed memory footprint per token, making transformer-based recognition practical on a single GPU.
- Reported training time is 3.7 minutes per epoch with 1.9M parameters, versus 5.0 minutes per epoch and 2.0M parameters for the SkateFormer baseline and 8.8 minutes per epoch for STTFormer, so the claimed efficiency is concrete on the NTU benchmarks.
- Accuracy remains within roughly one point of the best quadratic-attention transformer on each reported protocol, so the method is offered as an efficiency-accuracy trade-off rather than a pure accuracy gain.
Reading between the lines
- An implicit consequence of Eqs. (12)–(13) is that the HLA module contains no learned query/key/value projections, so the parameter savings relative to standard attention come partly from removing those projection matrices; the paper does not isolate this effect.
- Because $Q,K,V$ are just shifted copies of the input, the operation in Eq. (13) resembles a temporal shift-and-gate more than a similarity-based attention; testing this directly would require an ablation the paper does not report.
- The same HTC+HLA recipe could be transferred to other hierarchical sequence tasks such as motion prediction or point-cloud sequences, where the 'linear attention in hyperbolic space' claim would be testable in a new setting.
- The curvature choice $\kappa=-1$ is found by a small grid search on one dataset protocol; a learned or per-dataset curvature could shift the accuracy-efficiency frontier.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HyLiFormer, a skeleton-based action recognition transformer that first maps skeleton data into the Poincaré ball via a Hyperbolic Transformation with Curvatures (HTC) module and then applies a Hyperbolic Linear Attention (HLA) module claimed to reduce self-attention complexity from O(N^2) to O(N F^2). The central claims are that HLA is the first linear attention mechanism in the Poincaré model, that it provably keeps outputs in the Poincaré ball, and that it preserves accuracy while reducing computational cost. Experiments are reported on NTU RGB+D and NTU RGB+D 120, with ablations over curvature and comparisons to RWKV and Mamba baselines.
Significance. The motivation is relevant: efficient transformers for skeleton data are useful, and hyperbolic embeddings are a natural fit for hierarchical structure. The paper provides a clear problem statement, a full experimental section, and ablations over design choices. However, the significance rests entirely on the validity of Eqs. (12)-(13) as a linear-attention reformulation and on Lemma 4.2. As detailed below, these are not established; the HLA module as written is an undefined operation rather than an attention mechanism. With the central derivation unsound, the claimed efficiency-accuracy trade-off and the 'first hyperbolic linear attention' claim are not supported.
major comments (4)
- [Section 4.2, Eq. (12)] The query, key, and value matrices are defined in Eq. (12) as shifted slices of the raw input (Q=x[1:], K=x[2:], V=x[3:]) with no learned projection matrices. This is not 'consistent with the traditional self-attention mechanism' as stated, since Eq. (11) requires W_Q, W_K, W_V projections; with raw shifted inputs, the module cannot compute a learned similarity between tokens.
- [Section 4.2, Eq. (13)] Eq. (13) is asserted rather than derived from Eq. (11). It contains no index j, no summation, no normalization, and no definition of Sim(K_j^T V_j); the later remark that phi(x)=exp(x) is used as a kernel does not explain how this equation aggregates key-value pairs or avoids the N x N attention matrix. A charitable reading as global covariance, Q_i * sum_j K_j^T V_j, would require explicit projection, summation, and normalization that the paper does not provide, so the claimed O(N F^2) complexity is not demonstrated.
- [Section 4.2, Lemma 4.2] Lemma 4.2 is circular. The proof assumes that Sim(...) 'is designed such that it preserves hyperbolic distances and results in outputs bounded by (-1,1)' and that ||Q_i|| < -1/kappa, exactly the facts needed to conclude ||V_i|| < -1/kappa. No concrete Sim with the required properties is constructed, so the lemma does not provide a proof that the HLA output stays in the Poincaré ball.
- [Abstract, Section 4.2, Table 1] The complexity claim is internally inconsistent: the abstract and conclusion say O(N), while Section 4.2 says O(N F^2). More importantly, Table 1 shows HyLiFormer underperforms SkateFormer on X-Sub60 (91.7 vs 92.6) and X-Set120 (88.6 vs 89.3), so the claim of 'preserving model accuracy' is not supported without an ablation comparing to a quadratic-attention version of the same architecture.
minor comments (5)
- [Section 4.2] The notation for V is overloaded: V is used both for the value matrix in Eq. (12) and for the output in Eq. (13)-(14), which makes the derivation hard to follow.
- [Section 4.1, Eq. (10)] Eq. (10) writes x^B_kappa = -1/kappa * tanh(-kappa ||x||) * hat x, but the factor -1/kappa is positive for kappa < 0; the authors should clarify the sign convention or use a simpler form.
- [Section 4.1-4.2] The paper states 'It is rigorously proved' before both lemmas, but the proofs are informal and the second is circular; the wording should be adjusted to match the actual level of rigor.
- [Section 5.2] The training details are incomplete: no batch size, number of epochs, optimizer settings, or learning rate schedule are given, and the statement that 'the learning rate varies across datasets' is not quantified.
- [References and Table 1] Some references appear mismatched: the HyperFormer cited as [8] is a hypergraph transformer, while the hyperbolic vision transformer from [12] is a more relevant baseline for the claims; the comparison in Table 1 should be clarified.
Circularity Check
HLA's claimed linear attention reduces by construction: Eq. 13 is asserted with no derivation, Q/K/V are raw input shifts without learned projections, and Lemma 4.2 assumes the Sim boundedness it sets out to prove.
-
self definitional
[Section 4.2, Eq. (13) (with Eq. (12))]
"To avoid the explicit calculation of the similarity matrix QK^T in Eq. 11 and achieve linear attention, we propose an approximation by reordering the operations. The reformulated attention mechanism is given by V_i = Q_i · Sim(K_j^T · V_j)"
Eq. 11 defines self-attention with learned projections W_Q, W_K, W_V and a normalized summation over j. Eq. 13 is introduced as the 'reformulated attention mechanism' with no algebraic derivation: the index j, the summation, the normalization, and the kernel feature map are all undefined, while Eq. 12 defines Q, K, V as shifted slices of the raw input x, so no learned similarity remains. The claimed O(NF^2) complexity and attention semantics are therefore asserted by construction, not derived from Eq. 11.
-
self definitional
[Section 4.2, Lemma 4.2 proof]
"The similarity function Sim(K_j^T · V_j) is designed such that it preserves hyperbolic distances and results in outputs bounded by (−1, 1). Therefore, we have: ||V_i|| = ||Q_i · Sim(K_j^T · V_j)|| = ||Q_i||·||1|| < −1/κ."
The lemma claims to prove that the HLA output stays in the Poincaré ball, but the boundedness of Sim is exactly the design assumption used as the proof's premise: the proof writes ||V_i|| = ||Q_i||·||1||, thereby assuming Sim's output has unit norm and lies in (−1,1). No independent construction of Sim from Eq. 11 or from the later choice φ(x)=exp(x) is supplied, so the theorem restates its own hypothesis as the conclusion.
full rationale
The paper's experimental comparisons (Tables 1–4) are externally benchmarked on NTU RGB+D and NTU RGB+D 120, and the reported accuracy and training-time numbers are not circular. There is also no load-bearing self-citation: the same-group reference [25] appears only in related-work context. The circularity is concentrated in the theoretical derivation of the HLA module, which is the paper's central contribution. Eq. 12 defines Q, K, V as shifted raw input slices, so the asserted consistency with Eq. 11 is not established; Eq. 13 is labeled 'the reformulated attention mechanism' but omits the summation over j, the normalization, and any learned projection, making the claimed O(NF^2) reduction a property of the notation rather than a derived result. Lemma 4.2 then 'proves' the Poincaré-ball constraint by asserting that Sim is designed to produce bounded unit-norm outputs, so the conclusion is contained in the hypothesis. Because the claimed first hyperbolic linear attention and its complexity guarantee pass through these two steps, the derivation chain is partially circular: the mechanism is what it is by definition, and the defense of its core property assumes the property. The empirical results and the HTC module (Lemma 4.1, which is a direct verification of the tanh construction) remain independent, so the paper is not entirely vacuous. Overall score 7 reflects that the central theoretical claims reduce by construction while leaving external experiments intact.
Assumptions & free parameters
free parameters (1)
- Poincaré curvature kappa =
-1
assumptions (5)
- domain assumption Skeleton joint data has a tree-like hierarchical structure that is better captured in hyperbolic space than in Euclidean space.
- ad hoc to paper The kernel Sim(K_j^T * V_j) preserves hyperbolic distances and outputs scalars in the interval (-1, 1).
- ad hoc to paper Q = x[1:], K = x[2:], V = x[3:] with no learned projections is a valid instantiation of self-attention.
- domain assumption The unit vector computation x / ||x|| is defined for all skeleton points, including zero vectors.
- standard math The Poincaré model as defined by Eq. 1 with ||x|| < -1/kappa is the correct hyperbolic space for the subsequent operations.
Cite this review
Pith. "Pith review of HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition." pith.science (2026). https://pith.science/paper/25PBM4H7
@misc{pith2026250205869,
author = {Pith},
title = {Pith review of: HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/25PBM4H7}},
note = {Machine review of arXiv:2502.05869}
}
read the original abstract
Transformers have demonstrated remarkable performance in skeleton-based human action recognition, yet their quadratic computational complexity remains a bottleneck for real-world applications. To mitigate this, linear attention mechanisms have been explored but struggle to capture the hierarchical structure of skeleton data. Meanwhile, the Poincar\'e model, as a typical hyperbolic geometry, offers a powerful framework for modeling hierarchical structures but lacks well-defined operations for existing mainstream linear attention. In this paper, we propose HyLiFormer, a novel hyperbolic linear attention Transformer tailored for skeleton-based action recognition. Our approach incorporates a Hyperbolic Transformation with Curvatures (HTC) module to map skeleton data into hyperbolic space and a Hyperbolic Linear Attention (HLA) module for efficient long-range dependency modeling. Theoretical analysis and extensive experiments on NTU RGB+D and NTU RGB+D 120 datasets demonstrate that HyLiFormer significantly reduces computational complexity while preserving model accuracy, making it a promising solution for efficiency-critical applications.
Figures
Reference graph
Works this paper leans on
-
[1]
Dasom Ahn, Sangwon Kim, Hyunsu Hong, and Byoung Chul Ko. 2023. Star- transformer: a spatio-temporal cross attention transformer for human action recognition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision. 3330–3339
work page 2023
-
[2]
Ruwen Bai, Min Li, Bo Meng, Fengfa Li, Miao Jiang, Junxing Ren, and Degang Sun. 2022. Hierarchical graph convolutional skeleton transformer for action recognition. In 2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 01–06
work page 2022
-
[3]
Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. Multi-relational poincaré graph embeddings. Advances in Neural Information Processing Systems 32 (2019)
work page 2019
-
[4]
Jinghong Chen, Chong Zhao, Qicong Wang, and Hongying Meng. 2022. Hmanet: Hyperbolic manifold aware network for skeleton-based action recognition. IEEE Transactions on Cognitive and Developmental Systems 15, 2 (2022), 602–614
work page 2022
-
[5]
Weize Chen, Xu Han, Yankai Lin, Kaichen He, Ruobing Xie, Jie Zhou, Zhiyuan Liu, and Maosong Sun. 2024. Hyperbolic Pre-Trained Language Model.IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)
work page 2024
-
[6]
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu
-
[7]
Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, and Hanqing Lu. 2020. Skeleton-based action recognition with shift graph convolutional network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 183–192
work page 2020
-
[8]
Kaize Ding, Albert Jiongqian Liang, Bryan Perozzi, Ting Chen, Ruoxi Wang, Lichan Hong, Ed H Chi, Huan Liu, and Derek Zhiyuan Cheng. 2023. HyperFormer: Learning Expressive Sparse Feature Representations via Hypergraph Transformer. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2062–2066
work page 2023
Show all 39 references
-
[9]
Jeonghyeok Do and Munchurl Kim. 2025. Skateformer: skeletal-temporal trans- former for human action recognition. In European Conference on Computer Vision. Springer, 401–420
2025
-
[10]
Yong Du, Yun Fu, and Liang Wang. 2015. Skeleton based action recognition with convolutional neural network. In 2015 3rd IAPR Asian conference on pattern recognition (ACPR). IEEE, 579–583
2015
-
[11]
Yong Du, Wei Wang, and Liang Wang. 2015. Hierarchical recurrent neural net- work for skeleton based action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1110–1118
2015
-
[12]
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. 2022. Hyperbolic vision transformers: Combining improvements in metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7409–7419
2022
-
[13]
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. 2018. Hyperbolic neural networks. Advances in neural information processing systems 31 (2018)
2018
-
[14]
Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)
2023 arXiv
-
[15]
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. 2023. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF international conference on computer vision . 5961–5971
2023
-
[16]
Wenkai Huang, Yujia Yu, Haizhou Xu, Zhiwen Su, and Yu Wu. 2023. Hyperbolic music transformer for structured music generation. IEEE Access 11 (2023), 26893– 26905
2023
-
[17]
Chuankun Li, Yonghong Hou, Pichao Wang, and Wanqing Li. 2017. Joint distance maps based action recognition with convolutional neural networks. IEEE Signal Processing Letters 24, 5 (2017), 624–628
2017
-
[18]
Chuankun Li, Pichao Wang, Shuang Wang, Yonghong Hou, and Wanqing Li
-
[19]
Wenbo Li, Longyin Wen, Ming-Ching Chang, Ser Nam Lim, and Siwei Lyu. 2017. Adaptive RNN tree for large-scale human action recognition. In Proceedings of the IEEE international conference on computer vision . 1444–1452
2017
-
[20]
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In International confer- ence on machine learning . PMLR, 4055–4064
2018
-
[21]
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. 2024. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892 (2024)
2024 arXiv
-
[22]
Chiara Plizzari, Marco Cannici, and Matteo Matteucci. 2021. Skeleton-based action recognition via spatial and temporal transformer networks. Computer Vision and Image Understanding 208 (2021), 103219
2021
-
[23]
Chiara Plizzari, Marco Cannici, and Matteo Matteucci. 2021. Spatial temporal transformer network for skeleton-based action recognition. InPattern recognition. ICPR international workshops and challenges: virtual event, January 10–15, 2021, Proceedings, Part III. Springer, 694–701
2021
-
[24]
Helei Qiu, Biao Hou, Bo Ren, and Xiaohua Zhang. 2022. Spatio-temporal tuples transformer for skeleton-based action recognition.arXiv preprint arXiv:2201.02849 (2022)
2022 arXiv
-
[25]
Haoxuan Qu, Yujun Cai, and Jun Liu. 2024. Llms are good action recognizers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18395–18406
2024
-
[26]
Bin Ren, Mengyuan Liu, Runwei Ding, and Hong Liu. 2024. A survey on 3d skeleton-based action recognition using learning method. Cyborg and Bionic Systems 5 (2024), 0100
2024
-
[27]
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: a unified understanding of transformer’s attention via the lens of kernel. arXiv preprint arXiv:1908.11775 (2019)
2019 arXiv
-
[28]
A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)
2017
-
[29]
Pichao Wang, Wanqing Li, Chuankun Li, and Yonghong Hou. 2018. Action recognition based on joint trajectory maps with convolutional neural networks. Knowledge-Based Systems 158 (2018), 43–53
2018
-
[30]
Wenhan Wu, Ce Zheng, Zihao Yang, Chen Chen, Srijan Das, and Aidong Lu. 2024. Frequency Guidance Matters: Skeletal Action Recognition by Frequency-Aware Mixed Transformer. In Proceedings of the 32nd ACM International Conference on Multimedia. 4660–4669
2024
-
[31]
Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolu- tional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32
2018
-
[32]
Menglin Yang, Harshit Verma, Delvin Ce Zhang, Jiahong Liu, Irwin King, and Rex Ying. 2024. Hypformer: Exploring efficient transformer fully in hyperbolic space. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3770–3781
2024
-
[33]
Chengkun Zhang and Junbin Gao. 2021. Hype-han: Hyperbolic hierarchical attention network for semantic embedding. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 3990–3996
2021
-
[34]
Jiaxu Zhang, Yifan Jia, Wei Xie, and Zhigang Tu. 2022. Zoom transformer for skeleton-based group activity recognition. IEEE Transactions on Circuits and Systems for Video Technology 32, 12 (2022), 8646–8659
2022
-
[35]
Yuhan Zhang, Bo Wu, Wen Li, Lixin Duan, and Chuang Gan. 2021. STST: Spatial- temporal specialized transformer for skeleton-based action recognition. In Pro- ceedings of the 29th ACM International Conference on Multimedia . 3229–3237
2021
-
[36]
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. 2021. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision. 16259–16268
2021
-
[37]
Yuxuan Zhou, Xudong Yan, Zhi-Qi Cheng, Yan Yan, Qi Dai, and Xian-Sheng Hua. 2024. BlockGCN: Redefine Topology Awareness for Skeleton-Based Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2049–2058
2024
-
[2017]
In 2017 IEEE International conference on multimedia & expo workshops (ICMEW)
Skeleton-based action recognition using LSTM and CNN. In 2017 IEEE International conference on multimedia & expo workshops (ICMEW) . IEEE, 585– 590
2017
-
[2021]
In Proceedings of the IEEE/CVF international conference on computer vision
Channel-wise topology refinement graph convolution for skeleton-based action recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 13359–13368
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.