REVIEW 3 major objections 5 minor 42 references
STM: SpatioTemporal and Motion Encoding for Action Recognition
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that a 2D CNN with STM blocks can match or beat 3D and two-stream video models by encoding spatiotemporal and motion features together, without optical flow.
desk verdict Useful 2D efficient action-recognition block with genuinely strong temporal-dataset results, but the abstract's SOTA claim is contradicted by the paper's own tables and a missing SlowFast baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the STM block, a drop-in replacement for a ResNet residual block. Its CSTM half performs channel-wise temporal fusion: the feature tensor is reshaped to NHW x C x T, a 1D convolution with kernel size 3 mixes adjacent frames independently for each channel, and a 3x3 2D convolution follows. Its CMM half computes an approximate motion feature by applying a channel-wise 3x3 convolution to frame t+1 and subtracting frame t, with channel reduction by a factor of 16 before and restoration after; the T-1 difference maps are padded with zeros to keep temporal length. Summation fusion of the two branches plus an identity shortcut lets the block inherit ResNet's optimization and add only about 1.2% extra FLOPs over TSN. This machinery is what lets the paper claim that spatiotemporal and motion encoding need neither 3D kernels nor optical flow.
What would settle it
Run STM and the main baselines, such as TSM, ECO, and I3D, on Something-Something v1 under an identical inference protocol with the same number of temporal samples, same crops, and same score fusion, and check whether STM's top-1 advantage persists; a direct re-implementation that reproduces TSM's number under its own protocol but drops below it under STM's protocol would falsify the state-of-the-art claim.
Extended reading notes
Core claim
The central claim is that fusing two channel-wise operations—a temporal 1D convolution over the frame dimension and a frame-difference motion operator—inside a standard residual block produces features that carry both kinds of information. The CSTM reshapes the feature tensor so a 1D convolution runs along time independently per channel, then adds a 3x3 spatial convolution; the CMM reduces channels, convolves frame t+1, subtracts frame t, and concatenates the T-1 difference maps with a zero placeholder. Summing the two outputs and adding an identity shortcut gives the STM block. The authors claim that when every ResNet-50 residual block is replaced this way, the network reaches 49.2% top-1 on Something-Something v1 with 8 frames, 73.7% top-1 on Kinetics-400 with 16 frames, and does so with only about 1.2% more FLOPs than the TSN baseline.
Load-bearing premise
The state-of-the-art claim assumes that the accuracy numbers of the compared methods were obtained under the same evaluation protocol (number of sampled clips, crops, and score fusion), since STM uses 10 random temporal samples and three crops while many cited baselines use different settings.
Editorial extensions
If this is right
- If the reported gains hold under matched evaluation protocols, a single 2D RGB stream can replace the two-stream RGB-plus-optical-flow setup for temporal action recognition on Something-Something and Jester.
- STM with 8 frames outperforms I3D and ECO on Something-Something v1 while using 33.3G FLOPs versus 306G and 64G, pointing toward practical deployment of video models on a single GPU.
- On Kinetics-400, STM's top-5 accuracy of 91.6% is the best among the compared methods, including two-stream I3D, suggesting the spatiotemporal-plus-motion encoding also helps scene-related video classification.
- Because the STM block is a drop-in residual replacement, the same recipe should transfer to other ResNet-style backbones and related video tasks without redesigning the architecture.
Reading between the lines
- If the per-channel temporal convolution is the real source of gain, a testable extension is to apply the same reshape-and-convolve trick to other temporal fusion problems, such as temporal action localization, where expensive 3D backbones are currently standard.
- The CMM's zero-padding of the last time step means the network sees one fewer genuine motion frame; a circular padding or learned boundary handling might push accuracy slightly higher on longer clips, though the paper does not test this.
- A protocol-sensitive reading suggests part of the reported gap over TSM may come from the 10-random-sample, three-crop inference; reproducing both methods under identical sampling would separate algorithmic gain from evaluation gain.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an STM network for video action recognition that replaces ResNet residual blocks with STM blocks, each containing a Channel-wise SpatioTemporal Module (CSTM) and a Channel-wise Motion Module (CMM). CSTM applies channel-wise 1D temporal convolution to fuse spatiotemporal information, while CMM computes approximate motion features from adjacent frames via channel-wise 2D convolution and subtraction. The resulting 2D-CNN framework avoids 3D convolutions and pre-computed optical flow. The authors evaluate on Something-Something v1/v2, Jester, Kinetics-400, UCF-101, and HMDB-51, reporting strong results and claiming to outperform state-of-the-art methods on both temporal-related and scene-related datasets.
Significance. If the headline claim were true, STM would be an attractive efficient alternative to 3D CNNs and two-stream flow-based models, since it adds only about 1.2% FLOPs over the TSN baseline. The component-wise ablations in Section 4.5 provide useful evidence that both CSTM and CMM contribute to the accuracy gain over TSN on Something-Something. However, the paper's own tables contradict the unqualified state-of-the-art claim on multiple datasets, and several strong contemporary baselines (SlowFast, non-local networks) are cited but omitted from the comparisons. The core architectural idea is plausible and the efficiency numbers are interesting, but the paper as written overstates its conclusions and requires substantive revision before the claims can be accepted.
major comments (3)
- [Abstract and Section 4.4] The central claim that STM 'outperforms the state-of-the-art methods on both temporal-related datasets and scene-related datasets' is contradicted by the paper's own tables. On Kinetics-400 (Table 3), I3D Two-Stream achieves 74.2 top-1 versus STM's 73.7. On UCF-101 and HMDB-51 (Table 4), I3D Two-Stream achieves 98.0/80.7 versus STM's 96.2/72.2, and TSN Two-Stream achieves 97.0 on UCF-101 versus STM's 96.2. On Something-Something v1/v2 test sets (Table 1), TSM Two-Stream achieves 46.1/63.7 top-1 versus STM's 43.1/63.5. The conclusion in Section 5 repeats the unsupported claim. The authors should either restrict the claim to a clearly defined scope (e.g., efficient RGB-only 2D methods without optical flow) or add the missing baselines and revise the abstract accordingly.
- [Section 2 and Table 3] SlowFast is cited in Related Works but its results are absent from all comparison tables. The SlowFast paper (Feichtenhofer et al., 2018) reports top-1 accuracy on Kinetics-400 well above 77% with the 112-frame model, which is substantially higher than the 73.7% reported for STM. Non-local networks are also cited (reference [34]) and used to justify the inference protocol, but no non-local baseline appears in the experiments. Because the abstract claims state-of-the-art performance without qualification, the omission of these strong baselines is a load-bearing gap. The authors must add these comparisons or explicitly state that the claim is limited to the methods listed in their tables.
- [Section 4.2 (Inference) and Tables 1-4] The evaluation protocol comparability is not established. The paper uses 10 random temporal samples and three 256x256 crops resized to 224x224, following references [34,7], but many of the cited baselines were evaluated with different protocols (e.g., single center crop, 10 crops, varying temporal sampling and fusion schemes). The paper reports no error bars, no standard deviations, and no code release. Differences as small as 0.2-0.5 percentage points (e.g., STM 73.7 vs I3D Two-Stream 74.2 on Kinetics-400) are within typical run-to-run variability and cannot support a definitive superiority claim without controlled comparison. The authors should provide protocol details for all baselines, or at minimum clearly state that the comparison is approximate, and ideally report multiple runs.
minor comments (5)
- [Abstract] The word 'blcoks' should be 'blocks'.
- [Section 4.2] 'data argumentation' should be 'data augmentation'.
- [Section 4.5, Table 5] The text says CSTM brings 'about 28% top-1 accuracy improvement' and CMM brings '24.4% improvement' over TSN; since the top-1 accuracies are percentages, these should be described as percentage-point improvements (28.0 and 24.4 points, respectively), not relative percentages.
- [Figure 1 caption] 'Conv2 1 block' should likely be 'Conv2_1 block' for consistency with ResNet stage naming.
- [Section 4.4] The sentence 'Table 3 summaries the results' should be 'Table 3 summarizes the results'.
Circularity Check
No circularity: empirical benchmarks validate the proposed architecture; SOTA-claim issues are correctness, not circularity.
full rationale
This is an empirical architecture paper with no derivation chain that reduces to its inputs. The STM block is built from a Channel-wise SpatioTemporal Module (CSTM, Eq. 1) and a Channel-wise Motion Module (CMM, Eq. 2); both are defined directly on input feature maps with learnable kernels, and the design choices (summation vs concatenation, block placement, channel-wise vs ordinary convolution) are tested through ablations in Tables 5-8 rather than assumed. Reported accuracies are measured on held-out validation/test splits of independent benchmarks (Something-Something v1/v2, Jester, Kinetics-400, UCF-101, HMDB-51) against external baselines. No parameter is fitted to a subset of the benchmark and then the same quantity is renamed as a prediction; no uniqueness theorem from prior work by the authors is invoked; no load-bearing argument rests on a self-citation; and no known empirical pattern is merely renamed. The discrepancy between the abstract's claim that STM 'outperforms the state-of-the-art' on scene-related datasets and the paper's own tables (e.g., I3D Two-Stream reports 74.2 top-1 on Kinetics-400 versus STM's 73.7 in Table 3, and 98.0/80.7 on UCF-101/HMDB-51 versus STM's 96.2/72.2 in Table 4) is a correctness or comparison-protocol concern, not circularity. Under the required standard of exhibiting a specific reduction, no circular step is present, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- channel reduction ratio r =
16
- temporal kernel size =
3
- spatial kernel size =
3x3
- number of frames T =
8 or 16
assumptions (3)
- domain assumption Feature-level subtraction after a learned channel-wise convolution approximates optical flow motion.
- domain assumption Benchmark numbers from prior papers are comparable despite potential differences in evaluation protocols.
- domain assumption The datasets are correctly labeled and representative.
Cite this review
Pith. "Pith review of STM: SpatioTemporal and Motion Encoding for Action Recognition." pith.science (2026). https://pith.science/paper/F77KPFXE
@misc{pith2026190802486,
author = {Pith},
title = {Pith review of: STM: SpatioTemporal and Motion Encoding for Action Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/F77KPFXE}},
note = {Machine review of arXiv:1908.02486}
}
read the original abstract
Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.
Figures
Reference graph
Works this paper leans on
-
[34]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 7794–7803, 2018
work page 2018
-
[1]
In https://20bn.com/ datasets/jester
The 20bn-jester dataset v1. In https://20bn.com/ datasets/jester
-
[2]
Quo vadis, action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017
2017
-
[3]
Aˆ 2-nets: Double attention net- works
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. Aˆ 2-nets: Double attention net- works. In Advances in Neural Information Processing Sys- tems, pages 350–359, 2018
work page 2018
-
[4]
Spatio-temporal channel correlation networks for action classification
Ali Diba, Mohsen Fayyaz, Vivek Sharma, M Mahdi Arzani, Rahman Yousefzadeh, Juergen Gall, and Luc Van Gool. Spatio-temporal channel correlation networks for action classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 284–299, 2018
work page 2018
-
[5]
Flownet: Learning optical flow with convolutional networks
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Pro- ceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015
work page 2015
-
[6]
End-to-end learning of motion representation for video understanding
Lijie Fan, Wenbing Huang, Chuang Gan, Stefano Ermon, Boqing Gong, and Junzhou Huang. End-to-end learning of motion representation for video understanding. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6016–6025, 2018
work page 2018
-
[7]
Slowfast networks for video recognition
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. arXiv preprint arXiv:1812.03982, 2018
arXiv 2018
Show all 42 references
-
[8]
Spatiotemporal residual networks for video action recogni- tion
Christoph Feichtenhofer, Axel Pinz, and Richard Wildes. Spatiotemporal residual networks for video action recogni- tion. In Advances in neural information processing systems, pages 3468–3476, 2016
2016
-
[9]
Spatiotemporal multiplier networks for video action recog- nition
Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4768–4777, 2017
2017
-
[10]
Convolutional two-stream network fusion for video action recognition
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1933–1941, 2016
1933
-
[11]
The” something something” video database for learning and evaluating visual common sense
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...
2017
-
[12]
Stnet: Local and global spatial-temporal modeling for action recognition
Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Liming Wang, and Shilei Wen. Stnet: Local and global spatial-temporal modeling for action recognition. arXiv preprint arXiv:1811.01549, 2018
2018 arXiv
-
[13]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[14]
Flownet 2.0: Evolu- tion of optical flow estimation with deep networks
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolu- tion of optical flow estimation with deep networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2462–2470, 2017
2017
-
[15]
Batch normalization: Accelerating deep network training by reducing internal co- variate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In International Conference on Machine Learn- ing, pages 448–456, 2015
2015
-
[16]
Large-scale video classification with convolutional neural networks
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014
2014
-
[17]
Kuehne, H
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recog- nition. In Proceedings of the International Conference on Computer Vision (ICCV), 2011
2011
-
[18]
Motion feature network: Fixed motion filter for action recognition
Myunggi Lee, Seungeui Lee, Sungjoon Son, Gyutae Park, and Nojun Kwak. Motion feature network: Fixed motion filter for action recognition. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) , pages 387– 403, 2018
2018
-
[19]
Temporal shift module for efficient video understanding
Ji Lin, Chuang Gan, and Song Han. Temporal shift module for efficient video understanding. arXiv preprint arXiv:1811.08383, 2018
2018 arXiv
-
[20]
Learning spatio- temporal representation with pseudo-3d residual networks
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio- temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533–5541, 2017
2017
-
[21]
Optical flow estima- tion using a spatial pyramid network
Anurag Ranjan and Michael J Black. Optical flow estima- tion using a spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 4161–4170, 2017
2017
-
[22]
Two-stream con- volutional networks for action recognition in videos
Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In Ad- vances in neural information processing systems, pages 568– 576, 2014
2014
-
[23]
Ucf101: A dataset of 101 human actions classes from videos in the wild
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012
2012 arXiv
-
[24]
D3d: Distilled 3d networks for video ac- tion recognition
Jonathan C Stroud, David A Ross, Chen Sun, Jia Deng, and Rahul Sukthankar. D3d: Distilled 3d networks for video ac- tion recognition. arXiv preprint arXiv:1812.08249, 2018
2018 arXiv
-
[25]
Human action recognition using factorized spatio-temporal convolu- tional networks
Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E Shi. Human action recognition using factorized spatio-temporal convolu- tional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4597–4605, 2015
2015
-
[26]
Optical flow guided feature: a fast and ro- bust motion representation for video action recognition
Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: a fast and ro- bust motion representation for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1390–1399, 2018
2018
-
[27]
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InProceedings of the IEEE inter- national conference on computer vision , pages 4489–4497, 2015
2015
-
[28]
A closer look at spatiotemporal convolutions for action recognition
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recogni- tion, pages 6450–6459, 2018
2018
-
[29]
Long- term temporal convolutions for action recognition
G ¨ul Varol, Ivan Laptev, and Cordelia Schmid. Long- term temporal convolutions for action recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1510–1517, 2018
2018
-
[30]
Action recognition by dense trajectories
Heng Wang, Alexander Kl ¨aser, Cordelia Schmid, and Liu Cheng-Lin. Action recognition by dense trajectories. In CVPR 2011-IEEE Conference on Computer Vision & Pat- tern Recognition, pages 3169–3176. IEEE, 2011
2011
-
[31]
Action recognition with improved trajectories
Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE inter- national conference on computer vision , pages 3551–3558, 2013
2013
-
[32]
Appearance-and-relation networks for video classification
Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance-and-relation networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1430–1439, 2018
2018
-
[33]
Temporal segment net- works: Towards good practices for deep action recognition
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment net- works: Towards good practices for deep action recognition. In European conference on computer vision , pages 20–36. Springer, 2016
2016
-
[35]
Videos as space-time region graphs
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 399–417, 2018
2018
-
[36]
Spatiotemporal pyramid network for video action recog- nition
Yunbo Wang, Mingsheng Long, Jianmin Wang, and Philip S Yu. Spatiotemporal pyramid network for video action recog- nition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1529–1538, 2017
2017
-
[37]
Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 305–321, 2018
2018
-
[38]
A du- ality based approach for realtime tv-l 1 optical flow
Christopher Zach, Thomas Pock, and Horst Bischof. A du- ality based approach for realtime tv-l 1 optical flow. In Joint pattern recognition symposium , pages 214–223. Springer, 2007
2007
-
[39]
Recognize ac- tions by disentangling components of dynamics
Yue Zhao, Yuanjun Xiong, and Dahua Lin. Recognize ac- tions by disentangling components of dynamics. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6566–6575, 2018
2018
-
[40]
Temporal relational reasoning in videos
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Tor- ralba. Temporal relational reasoning in videos. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 803–818, 2018
2018
-
[41]
Mict: Mixed 3d/2d convolutional tube for human ac- tion recognition
Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, and Wenjun Zeng. Mict: Mixed 3d/2d convolutional tube for human ac- tion recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 449–458, 2018
2018
-
[42]
Eco: Efficient convolutional network for online video understanding
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional network for online video understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 695–712, 2018
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.