{"id":"3277e235-334f-4daf-b9f6-36b0a2f556b5","arxiv_id":"1908.02486","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A ResNet block that fuses channel-wise temporal convolution with learned feature-level motion subtraction achieves state-of-the-art accuracy on temporal video benchmarks.","lead":"This paper designs a 2D neural network block that combines spatiotemporal and motion features for video action recognition. The method reports top accuracy on several gesture and action benchmarks while avoiding expensive 3D convolutions and optical flow.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unqualified SOTA claim is contradicted by the paper's own tables and the cited-but-omitted SlowFast baseline.","rationale":"The STM architecture is a plausible and useful contribution: the CSTM/CMM design is lightweight, the gains over the TSN baseline on Something-Something and Jester are large, and the runtime comparison in Table 9 supports the efficiency argument. Those merits justify keeping the reader's CONDITIONAL verdict. However, the most load-bearing assertion is not primarily weak because of clip/crop protocol differences, as the reader's weakest_assumption suggests. It is weak because the paper's own tables already contain baselines that beat STM on scene-related datasets and on temporal-related test sets, and because the cited SlowFast work is missing from the comparisons. Protocol alignment would change gap sizes but would not remove the omitted baseline or the internal contradiction between the abstract and Section 4.4. The concrete check of adding SlowFast and retaining all two-stream rows settles the concern without needing new experiments. The verdict remains conditional: the paper should be published only after the abstract and SOTA claims are revised to match the evidence, or the missing baselines are added and evaluated under a common protocol.","tokens_in":13032,"tokens_out":9140,"duration_ms":97527,"concrete_test":"Construct the complete comparison table by taking Tables 3 and 4 and inserting the Kinetics-400 numbers reported in the cited SlowFast paper [7], while retaining the existing I3D Two-Stream and TSN Two-Stream rows, and also include the Something-Something v1/v2 test-set rows from Table 1. Check row by row whether STM's top-1 accuracy exceeds every entry. Since the paper's own tables already show two-stream baselines beating STM on Kinetics-400, UCF-101, HMDB-51, and the v1/v2 test sets, adding the missing SlowFast row is sufficient to settle that the unqualified outperforms-SOTA claim is false unless the claim is explicitly restricted to RGB-only 2D methods or to validation-set results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim requires that STM outperforms state-of-the-art methods on both temporal-related and scene-related datasets. The paper's own comparisons do not support this. In Table 3, I3D Two-Stream reports 74.2 top-1 on Kinetics-400 versus STM's 73.7; in Table 4, I3D Two-Stream scores 98.0/80.7 on UCF-101/HMDB-51 versus STM's 96.2/72.2, and TSN Two-Stream also exceeds STM on UCF-101. Even on the temporal-related test sets in Table 1, TSM Two-Stream achieves 46.1/63.7 top-1 on Something-Something v1/v2 versus STM's 43.1/63.5. The cited SlowFast work [7] is absent from all comparison tables despite reporting stronger Kinetics-400 accuracy. The load-bearing assumption is that flow-based methods and SlowFast can be excluded from the SOTA set, but the paper states no such restriction in the abstract. Section 4.4 itself concedes only competitive results on scene-related datasets, directly weakening the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an STM network for video action recognition that replaces ResNet residual blocks with STM blocks, each containing a Channel-wise SpatioTemporal Module (CSTM) and a Channel-wise Motion Module (CMM). CSTM applies channel-wise 1D temporal convolution to fuse spatiotemporal information, while CMM computes approximate motion features from adjacent frames via channel-wise 2D convolution and subtraction. The resulting 2D-CNN framework avoids 3D convolutions and pre-computed optical flow. The authors evaluate on Something-Something v1/v2, Jester, Kinetics-400, UCF-101, and HMDB-51, reporting strong results and claiming to outperform state-of-the-art methods on both temporal-related and scene-related datasets.","tokens_in":13274,"tokens_out":4478,"duration_ms":41819,"significance":"If the headline claim were true, STM would be an attractive efficient alternative to 3D CNNs and two-stream flow-based models, since it adds only about 1.2% FLOPs over the TSN baseline. The component-wise ablations in Section 4.5 provide useful evidence that both CSTM and CMM contribute to the accuracy gain over TSN on Something-Something. However, the paper's own tables contradict the unqualified state-of-the-art claim on multiple datasets, and several strong contemporary baselines (SlowFast, non-local networks) are cited but omitted from the comparisons. The core architectural idea is plausible and the efficiency numbers are interesting, but the paper as written overstates its conclusions and requires substantive revision before the claims can be accepted.","major_comments":[{"comment":"The central claim that STM 'outperforms the state-of-the-art methods on both temporal-related datasets and scene-related datasets' is contradicted by the paper's own tables. On Kinetics-400 (Table 3), I3D Two-Stream achieves 74.2 top-1 versus STM's 73.7. On UCF-101 and HMDB-51 (Table 4), I3D Two-Stream achieves 98.0/80.7 versus STM's 96.2/72.2, and TSN Two-Stream achieves 97.0 on UCF-101 versus STM's 96.2. On Something-Something v1/v2 test sets (Table 1), TSM Two-Stream achieves 46.1/63.7 top-1 versus STM's 43.1/63.5. The conclusion in Section 5 repeats the unsupported claim. The authors should either restrict the claim to a clearly defined scope (e.g., efficient RGB-only 2D methods without optical flow) or add the missing baselines and revise the abstract accordingly.","section":"Abstract and Section 4.4"},{"comment":"SlowFast is cited in Related Works but its results are absent from all comparison tables. The SlowFast paper (Feichtenhofer et al., 2018) reports top-1 accuracy on Kinetics-400 well above 77% with the 112-frame model, which is substantially higher than the 73.7% reported for STM. Non-local networks are also cited (reference [34]) and used to justify the inference protocol, but no non-local baseline appears in the experiments. Because the abstract claims state-of-the-art performance without qualification, the omission of these strong baselines is a load-bearing gap. The authors must add these comparisons or explicitly state that the claim is limited to the methods listed in their tables.","section":"Section 2 and Table 3"},{"comment":"The evaluation protocol comparability is not established. The paper uses 10 random temporal samples and three 256x256 crops resized to 224x224, following references [34,7], but many of the cited baselines were evaluated with different protocols (e.g., single center crop, 10 crops, varying temporal sampling and fusion schemes). The paper reports no error bars, no standard deviations, and no code release. Differences as small as 0.2-0.5 percentage points (e.g., STM 73.7 vs I3D Two-Stream 74.2 on Kinetics-400) are within typical run-to-run variability and cannot support a definitive superiority claim without controlled comparison. The authors should provide protocol details for all baselines, or at minimum clearly state that the comparison is approximate, and ideally report multiple runs.","section":"Section 4.2 (Inference) and Tables 1-4"}],"minor_comments":[{"comment":"The word 'blcoks' should be 'blocks'.","section":"Abstract"},{"comment":"'data argumentation' should be 'data augmentation'.","section":"Section 4.2"},{"comment":"The text says CSTM brings 'about 28% top-1 accuracy improvement' and CMM brings '24.4% improvement' over TSN; since the top-1 accuracies are percentages, these should be described as percentage-point improvements (28.0 and 24.4 points, respectively), not relative percentages.","section":"Section 4.5, Table 5"},{"comment":"'Conv2 1 block' should likely be 'Conv2_1 block' for consistency with ResNet stage naming.","section":"Figure 1 caption"},{"comment":"The sentence 'Table 3 summaries the results' should be 'Table 3 summarizes the results'.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-engineered 2D action-recognition block that does what it says on temporal datasets, but the abstract oversells it. The paper's own tables contradict the \"outperforms state-of-the-art\" claim on scene-related datasets, and the cited SlowFast baseline is missing from the comparisons.\n\nWhat's actually new: the STM block combines channel-wise 1D temporal convolution (CSTM) with a learned feature-level motion module (CMM) inside a standard ResNet. Neither module is revolutionary — TSM and MFNet are close relatives — but the specific combination, and the channel-wise choice, give a real accuracy/efficiency point. On Something-Something v1/v2 and Jester the numbers are genuinely strong: 8-frame STM beats TSM's 16-frame results on Something-Something v1, at similar FLOPs. The ablations are clean: each module contributes, summation beats concatenation, deeper placement helps, and channel-wise convolution beats ordinary convolution with fewer parameters. The runtime table is also useful and honestly framed.\n\nSoft spots. First, the abstract claims SOTA on both temporal-related and scene-related datasets, but Table 3 shows I3D Two-Stream at 74.2 vs STM's 73.7 on Kinetics-400, and Table 4 shows I3D Two-Stream at 98.0/80.7 on UCF-101/HMDB-51 vs 96.2/72.2. TSN Two-Stream also beats STM on UCF-101. Section 4.4 quietly says \"competitive,\" which is accurate. This is a load-bearing inconsistency between the abstract and the evidence. Second, SlowFast is cited in the intro but absent from all comparison tables; given its Kinetics-400 results, that omission matters for the SOTA claim. Third, no code, no error bars, and the evaluation protocol (10 random clips, 3 crops) may differ from some baselines, so the margins on scene-related sets could shift. These are fixable in revision; the temporal-dataset claims are better supported.\n\nWho it's for: anyone working on efficient video recognition, especially those building on TSM-style 2D temporal modules. It deserves a serious referee — the core empirical contribution is real and reproducible in principle — but I'd insist the authors either fix the abstract to match the tables or add the missing comparisons. If I were the editor, I'd send it out and ask for a revision that removes the SOTA claim on scene-related datasets unless SlowFast and non-local baselines are included.","headline":"Useful 2D efficient action-recognition block with genuinely strong temporal-dataset results, but the abstract's SOTA claim is contradicted by the paper's own tables and a missing SlowFast baseline.","tokens_in":13735,"tokens_out":1892,"would_cite":true,"duration_ms":17555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 2D CNN with STM blocks can match or beat 3D and two-stream video models by encoding spatiotemporal and motion features together, without optical flow.","keywords":["action recognition","spatiotemporal features","motion features","2D CNN","channel-wise convolution","optical flow-free","STM block","video understanding"],"falsifier":"Run STM and the main baselines, such as TSM, ECO, and I3D, on Something-Something v1 under an identical inference protocol with the same number of temporal samples, same crops, and same score fusion, and check whether STM's top-1 advantage persists; a direct re-implementation that reproduces TSM's number under its own protocol but drops below it under STM's protocol would falsify the state-of-the-art claim.","tokens_in":12882,"feed_emoji":"🎬","tokens_out":4495,"duration_ms":43527,"temperature":0.7,"pith_summary":"The paper tries to establish that a video action recognition network built from ordinary 2D convolutions can capture both the spatiotemporal structure and the motion of actions, without 3D convolutions or precomputed optical flow. It does this by inserting a new block, the STM block, into every residual block of a ResNet-50. The authors report that this single-stream 2D network outperforms prior state of the art on temporal-focused datasets such as Something-Something v1/v2 and Jester, and remains competitive with better top-5 accuracy on Kinetics-400, UCF-101, and HMDB-51. If correct, the result matters because it promises near-3D accuracy at close-to-2D cost, making video recognition more practical.","feed_headline":"A 2D-only video network matches 3D models without optical flow","feed_subtitle":"STM block encodes motion and spatiotemporal cues per channel; 8-frame Something-Something v1 accuracy reaches 49.2%.","key_machinery":"The load-bearing object is the STM block, a drop-in replacement for a ResNet residual block. Its CSTM half performs channel-wise temporal fusion: the feature tensor is reshaped to NHW x C x T, a 1D convolution with kernel size 3 mixes adjacent frames independently for each channel, and a 3x3 2D convolution follows. Its CMM half computes an approximate motion feature by applying a channel-wise 3x3 convolution to frame t+1 and subtracting frame t, with channel reduction by a factor of 16 before and restoration after; the T-1 difference maps are padded with zeros to keep temporal length. Summation fusion of the two branches plus an identity shortcut lets the block inherit ResNet's optimization and add only about 1.2% extra FLOPs over TSN. This machinery is what lets the paper claim that spatiotemporal and motion encoding need neither 3D kernels nor optical flow.","core_discovery":"The central claim is that fusing two channel-wise operations—a temporal 1D convolution over the frame dimension and a frame-difference motion operator—inside a standard residual block produces features that carry both kinds of information. The CSTM reshapes the feature tensor so a 1D convolution runs along time independently per channel, then adds a 3x3 spatial convolution; the CMM reduces channels, convolves frame t+1, subtracts frame t, and concatenates the T-1 difference maps with a zero placeholder. Summing the two outputs and adding an identity shortcut gives the STM block. The authors claim that when every ResNet-50 residual block is replaced this way, the network reaches 49.2% top-1 on Something-Something v1 with 8 frames, 73.7% top-1 on Kinetics-400 with 16 frames, and does so with only about 1.2% more FLOPs than the TSN baseline.","pith_inferences":["If the per-channel temporal convolution is the real source of gain, a testable extension is to apply the same reshape-and-convolve trick to other temporal fusion problems, such as temporal action localization, where expensive 3D backbones are currently standard.","The CMM's zero-padding of the last time step means the network sees one fewer genuine motion frame; a circular padding or learned boundary handling might push accuracy slightly higher on longer clips, though the paper does not test this.","A protocol-sensitive reading suggests part of the reported gap over TSM may come from the 10-random-sample, three-crop inference; reproducing both methods under identical sampling would separate algorithmic gain from evaluation gain."],"forward_implications":["If the reported gains hold under matched evaluation protocols, a single 2D RGB stream can replace the two-stream RGB-plus-optical-flow setup for temporal action recognition on Something-Something and Jester.","STM with 8 frames outperforms I3D and ECO on Something-Something v1 while using 33.3G FLOPs versus 306G and 64G, pointing toward practical deployment of video models on a single GPU.","On Kinetics-400, STM's top-5 accuracy of 91.6% is the best among the compared methods, including two-stream I3D, suggesting the spatiotemporal-plus-motion encoding also helps scene-related video classification.","Because the STM block is a drop-in residual replacement, the same recipe should transfer to other ResNet-style backbones and related video tasks without redesigning the architecture."],"supporting_citations":[{"why":"Temporal Segment Networks supplies the baseline training and inference recipe and the accuracy numbers that STM improves on.","marker":"[33]"},{"why":"TSM is the main 2D CNN competitor; STM compares accuracy, speed, and FLOPs against it.","marker":"[19]"},{"why":"I3D defines the Kinetics pre-training and the two-stream RGB-plus-flow baseline that STM aims to surpass without flow.","marker":"[2]"},{"why":"ResNet is the backbone whose residual blocks are replaced by STM blocks.","marker":"[13]"},{"why":"Something-Something v1 and v2 provide the primary temporal-relation benchmarks for the main comparisons.","marker":"[11]"},{"why":"ECO is an efficient video model used in runtime and FLOPs comparisons.","marker":"[42]"},{"why":"MFNet proposes fixed motion filters; CMM is positioned as learning channel-wise feature-level motion instead.","marker":"[18]"},{"why":"TRN is a 2D temporal relational reasoning baseline on Something-Something that STM outperforms.","marker":"[40]"}],"fun_headline_variants":["2D action net uses channel-wise motion, matches 3D without flow","STM: a 2D block for both motion and spatiotemporal encoding","No optical flow: 2D ResNet with STM hits 49.2% on Sth-Sth","Unified 2D framework: STM block combines motion and temporal cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art claim assumes that the accuracy numbers of the compared methods were obtained under the same evaluation protocol (number of sampled clips, crops, and score fusion), since STM uses 10 random temporal samples and three crops while many cited baselines use different settings.","fun_headline_variants_meta":{"raw":{"variants":["2D action net uses channel-wise motion, matches 3D without flow","STM: a 2D block for both motion and spatiotemporal encoding","No optical flow: 2D ResNet with STM hits 49.2% on Sth-Sth","Unified 2D framework: STM block combines motion and temporal cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000374,"raw_usage":{"total_tokens":1992,"prompt_tokens":939,"completion_tokens":1053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":963}},"tokens_in":555,"tokens_out":1053,"duration_ms":10229,"temperature":1.0,"reasoning_tokens":963,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:41:51.351238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run STM and the main baselines, such as TSM, ECO, and I3D, on Something-Something v1 under an identical inference protocol with the same number of temporal samples, same crops, and same score fusion, and check whether STM's top-1 advantage persists; a direct re-implementation that reproduces TSM's number under its own protocol but drops below it under STM's protocol would falsify the state-of-the-art claim.","supporting_citations":[{"cited_title":"Temporal segment net- works: Towards good practices for deep action recognition","cited_arxiv_id":null,"evidence_quote":"Temporal Segment Networks supplies the baseline training and inference recipe and the accuracy numbers that STM improves on."},{"cited_title":"The” something something” video database for learning and evaluating visual common sense","cited_arxiv_id":null,"evidence_quote":"Something-Something v1 and v2 provide the primary temporal-relation benchmarks for the main comparisons."},{"cited_title":"Eco: Efficient convolutional network for online video understanding","cited_arxiv_id":null,"evidence_quote":"ECO is an efficient video model used in runtime and FLOPs comparisons."},{"cited_title":"Motion feature network: Fixed motion filter for action recognition","cited_arxiv_id":null,"evidence_quote":"MFNet proposes fixed motion filters; CMM is positioned as learning channel-wise feature-level motion instead."},{"cited_title":"Temporal relational reasoning in videos","cited_arxiv_id":null,"evidence_quote":"TRN is a 2D temporal relational reasoning baseline on Something-Something that STM outperforms."}],"review_version":1}