REVIEW 3 major objections 5 minor 17 references
Efficient Egocentric Action Recognition with Multimodal Data
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that egocentric action recognition can cut CPU usage by about 3x by feeding hand pose at 30 Hz while dropping RGB to 10 Hz, with nearly no loss in F1-score.
desk verdict A useful but under-supported empirical trade-off map: the claimed 3x CPU saving at matched accuracy needs error bars and on-device validation before it can be relied on. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a two-stream 'Multimodal Temporal MLP' (MM-TMLP): an RGB stream built on a LeViT vision-transformer feature extractor followed by a Temporal MLP, and a hand-pose stream built on an MLP feature extractor paired with a second Temporal MLP; the final-time-step outputs of the two streams are concatenated and classified. The mechanism doing the work is the relative sampling frequency of the two modalities. Because the RGB stream is the computationally expensive branch and the hand-pose stream is cheap, holding hand pose at a high rate while downsampling RGB shifts the compute budget without losing the motion cues that hand keypoints provide.
What would settle it
Measure inference time and CPU usage of the same two-stream models on an actual wearable or embedded CPU, and compare the 30 Hz RGB + 30 Hz hand pose configuration with the 10 Hz RGB + 30 Hz hand pose configuration; if the latter does not show roughly a 3x CPU reduction at near-equal macro F1, the paper's central efficiency claim is not portable to its target hardware. A second check is reproducing the frequency sweep on the test set to confirm the F1 plateau at 10 Hz RGB.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a measured trade-off curve: for a fixed hand-pose frequency, lowering the RGB sampling rate causes a gradual, not catastrophic, F1 decline, while CPU usage falls roughly threefold with each threefold reduction in RGB frequency; and for a fixed RGB frequency, adding the hand-pose stream improves F1 regardless of its sampling rate. The headline configuration, 10 Hz RGB with 10–30 Hz hand pose, reaches nearly the same macro F1-score as the fully sampled 30 Hz RGB + 30 Hz hand pose model while reducing CPU usage by approximately 3x. This is presented as evidence that multimodal input strategies, with frequency as a design lever, make efficient real-time egocentric action recognition feasible on resource-constrained wearable devices.
Load-bearing premise
The load-bearing premise is that CPU usage measured on a single thread of a server CPU faithfully represents the relative computational cost of the RGB and hand-pose streams on the wearable device where the system is meant to run.
Editorial extensions
If this is right
- A 10 Hz RGB stream with 10–30 Hz hand pose can replace a 30 Hz RGB + 30 Hz hand pose setup on resource-limited hardware with nearly unchanged macro F1.
- CPU usage scales roughly threefold with each threefold change in RGB sampling frequency, so deployment can pick an operating point on the accuracy–cost curve.
- For any fixed RGB frequency, adding hand pose improves F1 regardless of hand-pose sampling frequency, so the hand-pose branch is a reliable accuracy supplement.
- RGB-only sequence models lose accuracy more sharply as RGB frequency drops than multimodal models do, indicating that hand pose cushions the loss.
- Modality-aware sampling is a practical alternative to model compression for reducing on-device CPU cost.
Reading between the lines
- If the CPU-cost ratios transfer to mobile CPUs, the same sampling lever could let other high-cost modalities (e.g., higher-resolution RGB or optical flow) be downsampled while a cheap modality such as gaze, head pose, or audio carries the temporal signal—an extension the paper does not test.
- The 3x CPU gain is measured on a server CPU single thread, not on the target wearable; an on-device benchmark is the natural next experiment, since the relative cost of the ViT versus the hand-pose MLP may differ on embedded hardware.
- Because the effect is about a cheap stream compensating for a slow expensive stream, the result may generalize beyond egocentric action recognition to any multimodal sequence task with heterogeneous per-modality costs, though that generalization is speculative.
- A direct testable extension: on the same dataset, replacing hand pose with an equally cheap modality (e.g., head-gaze keypoints) and re-running the frequency sweep would show whether the compensation is specific to hand motion or general to any auxiliary stream.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical study of egocentric action recognition (EAR) on the H2O dataset using a two-stream architecture: an RGB stream based on a LeViT-256-distilled feature extractor with a Temporal MLP, and a 3D hand-pose stream based on an MLP feature extractor with a Temporal MLP. The authors systematically vary the sampling frequencies of RGB (30/10/3/1 Hz) and hand pose (30/10/3/1 Hz) inputs, measure macro F1-score and CPU usage on a single thread of an AMD EPYC 7742, and report that configurations with 10 Hz RGB and 10–30 Hz hand pose achieve nearly the same F1-score as the full 30 Hz RGB + 30 Hz hand pose setup while reducing CPU usage by approximately 3×. The paper concludes that modality-aware sampling rates are a promising efficiency lever for real-time EAR on resource-constrained XR devices.
Significance. If the central trade-off claim is reliable, the paper identifies a simple and actionable design insight: keeping the cheap hand-pose stream at high frequency while downsampling the expensive RGB stream can preserve accuracy while cutting computational cost. The study is clearly motivated by a real deployment constraint (Magic Leap 2) and the plotted trend in Figure 2 is internally consistent: RGB frequency dominates CPU usage, and the hand-pose stream mitigates accuracy loss at low RGB rates. The paper is also commendably concrete in reporting CPU usage per configuration, which makes the efficiency trend directly visible. However, the significance is currently limited by methodological weaknesses: the accuracy-preservation half of the claim rests on single-run point estimates with no error characterization, and the CPU numbers come from a server CPU rather than the target XR device. These issues are fixable within the manuscript's scope, but they must be addressed before the central claim can be considered established.
major comments (3)
- [§3.3, Figure 2] The headline claim that configurations with 10 Hz RGB and 10–30 Hz hand pose achieve 'nearly the same F1-score' as the full 30/30 setup is supported only by single point estimates in Figure 2. The paper reports no multiple seeds, standard deviations, confidence intervals, or significance tests, and it does not state the size of the H2O validation set used. On a small validation set, run-to-run training variance can easily exceed the reported accuracy gap, so the 'minimal to no loss' statement is not currently a measurable claim. Please report the mean and variance over at least three to five runs, or otherwise provide a statistical characterization (e.g., confidence intervals for the macro F1 difference), and state the validation set size and label distribution.
- [§2.2, Figure 2 caption] CPU usage is measured on a single thread of an AMD EPYC 7742 server CPU, yet the paper motivates the study by deployment on Magic Leap 2, which has a mobile/embedded CPU with different relative costs for a ViT (LeViT) versus an MLP hand-pose model. The reported 3× CPU reduction depends on the assumption that RGB feature extraction dominates the total cost on the target hardware. Because the paper does not validate this proxy on-device, the practical efficiency claim may not transfer to the intended platform. Please either include on-device measurements (even a small subset) or add a clear discussion of how the relative CPU cost might differ on embedded CPUs and how that would affect the reported trade-off.
- [§2.1–§2.4] The manuscript omits many details needed to reproduce the experiments: training hyperparameters (number of epochs, batch size, optimizer, learning rate schedule), GPU data-parallel setup, the exact mechanism for reducing sampling frequencies (frame skipping versus retraining with shorter sequences), the hand-pose keypoint representation (which keypoints, how many, coordinate frame, and whether they are ground-truth or predicted), and the H2O train/validation split and preprocessing specifics. For an empirical study whose conclusions are quantitative trends, these omissions make it impossible for a reader to assess the robustness of the results or to reproduce the curves in Figure 2. Please add a dedicated experimental-details section or supplementary material addressing these points.
minor comments (5)
- [Figure 2 caption] The caption states 'over a one-second input window,' but Section 3 says all sequence models were trained on 2-second sequences (60 time steps at 30 Hz). Please clarify whether CPU usage is measured per second of input or per full sequence, and ensure consistent wording.
- [§3.3] The sentence 'For a fixed fHP, the F1-score declines more sharply as fRGB decreases, though the hand pose stream helps mitigate this drop' is confusing: if the hand-pose stream mitigates the drop, the decline should be less sharp than in the RGB-only case, not more. Please rephrase to compare the multimodal model against the RGB-only model explicitly.
- [§3.1] The F1 scores of the single-frame models (RegNet, LeViT-256, LeViT-256-distilled, HP-MLP, FusionNet) are not reported numerically; only qualitative statements are given. Include a table with the exact macro F1 and CPU usage values for these models so readers can contextualize the sequence-model results.
- [§2.4] The sentence 'only shared augmentations are applied' is vague. Specify which augmentations are considered shared (e.g., random cropping, flipping, rotation) and which are modality-specific, as this affects the cross-modal consistency claim.
- [§3] The paper uses 'macro F1-score' but does not define it; please state that it is the unweighted average of per-class F1 scores, since the class distribution may be imbalanced in the H2O dataset.
Circularity Check
No significant circularity: the paper's claims are direct empirical measurements on an external benchmark.
full rationale
This paper is an empirical study of sampling-frequency trade-offs in egocentric action recognition. The central claim, that configurations using 10 Hz RGB and 10-30 Hz hand pose achieve nearly the same F1-score as the full 30 Hz RGB and 30 Hz hand pose setup while reducing CPU usage by about 3x, is a direct reading of measured macro F1 scores and CPU usage on the H2O benchmark. There are no fitted parameters whose values encode the reported outcome, no uniqueness theorem imported from prior work, and no self-citation chain supporting the main result. The model architecture choices, such as LeViT-256-distilled and the Temporal MLP, are justified by separate single-frame experiments and are not used to derive the frequency trade-off. The absence of error bars and the server-CPU proxy are legitimate reliability and generalization concerns, but they are not circularity: the conclusions are not equivalent to the inputs by construction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Temporal MLP hyperparameters
- Training hyperparameters
- Sequence window length =
2 seconds (60 time steps at 30 Hz)
- Distillation setup for LeViT-256-distilled
assumptions (3)
- domain assumption H2O dataset labels are correct and the frame-level labeling strategy (assigning each frame the action label of its segment) is a valid supervision signal.
- domain assumption The LeViT-256-distilled and Temporal MLP feature extractors are adequate for the egocentric action recognition task.
- domain assumption Single-thread CPU inference on an AMD EPYC 7742 is a representative proxy for the computational cost on the target XR device (Magic Leap 2).
Cite this review
Pith. "Pith review of Efficient Egocentric Action Recognition with Multimodal Data." pith.science (2026). https://pith.science/paper/IXWNMQGN
@misc{pith2026250601757,
author = {Pith},
title = {Pith review of: Efficient Egocentric Action Recognition with Multimodal Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/IXWNMQGN}},
note = {Machine review of arXiv:2506.01757}
}
read the original abstract
The increasing availability of wearable XR devices opens new perspectives for Egocentric Action Recognition (EAR) systems, which can provide deeper human understanding and situation awareness. However, deploying real-time algorithms on these devices can be challenging due to the inherent trade-offs between portability, battery life, and computational resources. In this work, we systematically analyze the impact of sampling frequency across different input modalities - RGB video and 3D hand pose - on egocentric action recognition performance and CPU usage. By exploring a range of configurations, we provide a comprehensive characterization of the trade-offs between accuracy and computational efficiency. Our findings reveal that reducing the sampling rate of RGB frames, when complemented with higher-frequency 3D hand pose input, can preserve high accuracy while significantly lowering CPU demands. Notably, we observe up to a 3x reduction in CPU usage with minimal to no loss in recognition performance. This highlights the potential of multimodal input strategies as a viable approach to achieving efficient, real-time EAR on XR devices.
Figures
Reference graph
Works this paper leans on
-
[1]
Pratyusha Das and Antonio Ortega. Symmetric sub-graph spatio-temporal graph convolution and its application in complex activity recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3215–3219. IEEE, 2021. 1
work page 2021
-
[2]
Avatars grow legs: Generating smooth human motion from sparse tracking in- puts with diffusion model
Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking in- puts with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 481–490, 2023. 2
work page 2023
-
[3]
Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with rolling- unrolling lstms and modality attention. In Proceedings of the IEEE/CVF International conference on computer vision, pages 6252–6261, 2019. 1
work page 2019
-
[4]
Levit: a vision transformer in convnet’s clothing for faster inference
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv ´e J ´egou, and Matthijs Douze. Levit: a vision transformer in convnet’s clothing for faster inference. In Proceedings of the IEEE/CVF interna- tional conference on computer vision , pages 12259–12269,
-
[5]
Distill- ing the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 3
arXiv 2015
-
[6]
Georgios Kapidis, Ronald Poppe, Elsbeth van Dam, Lu- cas PJJ Noldus, and Remco C Veltkamp. Object detection- based location and activity classification from egocentric videos: A systematic analysis. Smart Assisted Living: To- ward An Open Smart-Home Infrastructure , pages 119–145,
-
[7]
H2o: Two hands manipulating objects for first person interaction recognition
Taein Kwon, Bugra Tekin, Jan St ¨uhmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10138–10148, 2021. 2
work page 2021
-
[8]
Yin Li, Miao Liu, and James M. Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European Conference on Computer Vi- sion (ECCV), 2018. 1
work page 2018
Show all 17 references
-
[9]
Detecting activi- ties of daily living in first-person camera views
Hamed Pirsiavash and Deva Ramanan. Detecting activi- ties of daily living in first-person camera views. In 2012 IEEE conference on computer vision and pattern recogni- tion, pages 2847–2854. IEEE, 2012. 1
2012
-
[10]
Designing network design spaces
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Designing network design spaces. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10428–10436,
-
[11]
On the utility of 3d hand poses for action recognition
Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. On the utility of 3d hand poses for action recognition. InEuropean Conference on Computer Vision, pages 436–454. Springer, 2025. 1
2025
-
[12]
Convolutional lstm network: A machine learning approach for precipitation nowcasting
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing sys- tems, 28, 2015. 1
2015
-
[13]
Two-stream con- volutional networks for action recognition in videos
Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. Ad- vances in neural information processing systems , 27, 2014. 1
2014
-
[14]
Convolutional long short-term memory networks for recognizing first per- son interactions
Swathikiran Sudhakaran and Oswald Lanz. Convolutional long short-term memory networks for recognizing first per- son interactions. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, 2017. 1
2017
-
[15]
Attention is all we need: Nailing down object-centric attention for egocen- tric activity recognition
Swathikiran Sudhakaran and Oswald Lanz. Attention is all we need: Nailing down object-centric attention for egocen- tric activity recognition. arXiv preprint arXiv:1807.11794 ,
-
[16]
Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world
Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bu- gra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Polle- feys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world...
2023
-
[17]
Spatial tempo- ral graph convolutional networks for skeleton-based action recognition
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial tempo- ral graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on arti- ficial intelligence, 2018. 1 4
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.