Three Branches: Detecting Actions With Richer Features

Cewu Lu; Jiajun Tang; Jin Xia

REVIEW 3 major objections 4 minor 22 references

Three Branches: Detecting Actions With Richer Features

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A three-branch fusion of clip-level, person-level, and long-term features reports 32.49% mAP on AVA action localization, beating every 2018 AVA submission by over 10 points.

desk verdict A credible challenge report with a strong AVA number, but the missing same-pretraining ablations leave the three-branch contribution unproven. read the letter →

arxiv 1908.04519 v1 pith:XX27XQQH submitted 2019-08-13 cs.CV

Jin Xia , Jiajun Tang , Cewu Lu This is my paper · ORCID

classification cs.CV

keywords spatio-temporalactionlocalizationAVAdatasetSlowFastnetworkslong-termfeaturebanksrecognitionfocallossglobalcontextKinetics-700

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that fusing three sources of information—global video-clip features, short-term person-specific region features, and long-term feature banks of past activity—improves spatio-temporal action localization on the AVA benchmark. The authors report that their fused model reaches 32.49% mAP on the AVA test set, more than 10 points above all submissions to the same challenge the previous year. They also report a 21.59% average top-1/top-5 error on Kinetics-700 after pretraining their SlowFast backbone on Kinetics-600 and fine-tuning on Kinetics-700. A careful reader would care because this is evidence, if the attribution holds, that combining complementary temporal scales and global scene context is a useful direction for fine-grained action detection.

What carries the argument

The load-bearing mechanism is the three-branch fusion around a SlowFast backbone, together with two auxiliary training signals. SlowFast provides the clip-level feature maps from slow (8-frame) and fast (32-frame) pathways; 3D RoI pooling, implemented by repeating 2D RoI pooling along time, turns detection boxes into short-term person features; LFB blocks attend to long-term person features from neighboring seconds. A per-class sigmoid branch on the global clip feature defines the global loss, and a focal/softmax loss handles multi-label and pose-class imbalance. These components are concatenated before the final classifier, and the paper reports that concatenation beats summation.

What would settle it

A controlled experiment would settle it: fine-tune the same SlowFast backbone, initialized from the same Kinetics-700 weights, on AVA using the paper's training recipe, once with only the backbone features and once with the full three-branch fusion and global loss. If the backbone-only model already reaches roughly 32% mAP, the three-branch design is not the source of the improvement.

Watch

Extended reading notes

Core claim

The central claim is that a three-branch architecture for AVA action detection beats the previous best submissions by a wide margin. The first branch extracts global features of the entire video clip with a SlowFast 3D CNN; the second branch uses 3D RoI pooling to crop short-term person features from those clip features according to human boxes; the third branch supplies long-term person features through LFB-style attention blocks over features sampled across 61 seconds. The three feature types are concatenated and fed to a per-person classifier, with a scene-level global loss that predicts all actions in the clip. On this design the paper reports 32.49% mAP on AVA test, compared with 27.20% for LFB and 27.10% for SlowFast alone, and it argues that the combination, not any single component, is what produces the gain.

Load-bearing premise

The load-bearing premise is that the reported gain over prior AVA results is caused by the three-branch design rather than by the stronger backbone and larger Kinetics-700 pretraining.

Editorial extensions

If this is right

On the AVA test set, the full model reaches 32.49% mAP, a gain of more than 10 points over every AVA challenge submission from 2018.
Pretraining the SlowFast backbone on Kinetics-600 and fine-tuning on Kinetics-700 yields a 21.59% average top-1/top-5 error on Kinetics-700.
Concatenating short-term person features, long-term feature-bank features, and global clip features works better than summing the person and long-term streams, according to the paper's comparison.
A scene-level global loss adds accuracy even when the backbone already uses dilated res5 filters and Non-Local blocks, the paper reports.
Using softmax loss for pose classes and focal loss for unbalanced action classes improves training over plain per-class sigmoid loss, the paper argues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that its headline table mixes backbones and pretraining data, so the 10-point gain is an upper bound on the three-branch design's contribution until a matched ablation is run.
A testable extension is to apply the same three-branch fusion and global scene loss to spatio-temporal datasets with different action vocabularies, such as Charades or ActivityNet, where scene-person interactions also matter.
Because the paper reports that Transformer blocks and more than two LFB blocks give no gain, the effective part of the long-term branch may be simply the availability of long-range person features plus the global loss, rather than learned attention.
The global loss could serve as weak supervision: it needs only clip-level action labels, so it may transfer to settings where per-person spatio-temporal annotations are unavailable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi [Name],

Quick take: this is a credible CVPR 2019 challenge report, not a method paper. The 32.49% mAP on AVA is a real number from a real system, and the three-branch fusion (global clip, short-term RoI, long-term feature bank) is a sensible engineering composition. But the paper never isolates the contribution of those branches, and the comparison table mixes backbones and pretraining datasets. So the central claim that richer features are what drive the gain is unsupported as written.

What the paper does well: the recipe is described carefully enough to reproduce the system in principle — detection on key frames, 5-second clips, 61-second feature bank context, two LFB blocks, concatenation over summation, global scene loss, softmax pose loss, focal loss, multi-scale test augmentation. The authors also report a negative result (Transformer blocks didn't help), which is the kind of honesty you want to see in a challenge write-up. The Kinetics-700 result (21.59% error) is a useful data point.

The soft spots are exactly what the stress-test note says. Table 2 compares against LFB (K400 pretraining, 27.20), Action Transformer (K400, 24.93), and SlowFast (K600, 27.10) — none controlled for the same backbone and pretraining. The 32.49 mAP gap over the closest baseline could be largely explained by Kinetics-700 pretraining plus dilated res5, focal loss, or test augmentation. The paper states qualitatively that the global loss is 'still beneficial' and that concatenation beats sum, but gives no mAP numbers for any ablation. That's a missing-support flag. For a workshop report this might be acceptable, but for a peer-reviewed paper claiming a 10-point improvement, you need at least one same-pretraining ablation.

Also note: the abstract's claim about outperforming all CVPR 2018 submissions is a time-relative statement, not a controlled comparison. It's true but less impressive than a head-to-head under identical training conditions.

Bottom line: this paper has a useful empirical result and a clear description, but the evidence is not there to attribute the gain to the proposed fusion. It's worth a referee's time for a workshop or short-paper track, but as a full conference paper it needs ablations. I'd engage with it as a baseline reference, but I wouldn't cite it as evidence that three branches help.

Referee Report

3 major / 4 minor

Summary. The manuscript describes a three-branch model for the CVPR 2019 Activity Recognition challenge. For spatio-temporal action localization on AVA, it extracts (i) global clip features from a SlowFast backbone, (ii) short-term person features via 3D RoI pooling on detected human boxes, and (iii) long-term person features from a feature bank processed by LFB blocks, and concatenates them for final classification. A global scene-level classification loss, a pose-specific softmax loss, and focal loss are also used. The authors report 21.59% average top-1/top-5 error on Kinetics-700 and 32.49% mAP on AVA test, claiming this outperforms all CVPR 2018 AVA submissions by more than 10 mAP. The paper is a challenge report; no code, trained models, or official evaluation outputs are released.

Significance. The reported numbers, if reproducible, would represent a substantial empirical advance for spatio-temporal action localization on AVA: 32.49% mAP versus 27.20% for LFB and 27.10% for SlowFast with Kinetics-600 pretraining, plus a competitive 21.59% error rate on Kinetics-700. The paper is a useful systems report that clearly describes a plausible three-branch architecture and several practical training choices. Its scientific significance is currently limited because the design is not isolated from confounds: without a controlled ablation, the reader cannot tell which component or pretraining change produces the gain. The work would become significant if the authors add same-protocol ablations and release code or evaluation outputs.

major comments (3)

[§4, Table 2] The paper's central claim—that fusing global clip, short-term RoI, and long-term feature-bank branches yields the reported 32.49% mAP—is not supported by the comparisons in Table 2. The row labeled 'Ours' uses SlowFast with Kinetics-700 pretraining, whereas SlowFast is evaluated with Kinetics-600, and LFB and Action Transformer use Kinetics-400. None of these rows is a same-backbone, same-pretraining control without the three-branch fusion. The 5.39-point gap over the SlowFast row could therefore be explained by the larger pretraining set, dilated res5, the pose-specific softmax loss, focal loss, or multi-scale test augmentation. I request an ablation that keeps the backbone, pretraining, and losses fixed and removes/adds the three branches and the global loss, reporting validation and test mAP for each configuration.
[§3, 'Global Loss' and 'Long-Term Feature Operation'] Several design choices are asserted without quantitative evidence: the global loss is 'still beneficial,' concatenation is better than summation, two LFB blocks are sufficient, and dropout with dimension reduction reduces overfitting. None of these statements is accompanied by a numeric ablation or a table. Because the paper's title and abstract attribute the gain to the three-branch architecture, at least one ablation isolating each branch (global, short-term RoI, long-term LFB) and the global loss is required. As written, the qualitative statements are not verifiable from the manuscript.
[§4, Experiments] The experimental section does not report variance or repeated-run statistics, and the entire comparison rests on single test-set numbers. For the AVA task this is partly dictated by the challenge's official evaluation, but the manuscript should at least provide validation mAP for the different configurations and state how many runs produced the final numbers. Without this, it is unclear whether the 5.39 mAP advantage over the comparable SlowFast row is larger than run-to-run variation.

minor comments (4)

[Abstract, §1, §5] Typos and an incomplete sentence should be corrected: 'wole' should be 'whole,' 'futher' should be 'further,' 'nerual' should be 'neural,' 'phrase' should be 'phase,' and the sentence in §5 beginning 'Key point feature extracted by well developed model [22] can be a for action recognition' is missing a word.
[Table 1] Table 1 lists 'Pretrain Kinetics-600' while the text says the model is trained on Kinetics-600 and then fine-tuned on Kinetics-700; please clarify whether the reported 21.59% error is from a model fine-tuned on Kinetics-700 or from a model pretrained only on Kinetics-600.
[§3, 'Long-Term Feature Operation'] The choice of 305 long-term person features (5 persons per second for 61 seconds) is stated without justification or ablation, even though this parameter directly controls the memory footprint and temporal receptive field of the LFB branch.
[§4, Experiments] The human-detector confidence threshold of 0.8 is stated without analysis; a short sensitivity check would be helpful because detection quality directly affects AVA mAP.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical benchmark scores rather than deriving them from fitted inputs or self-citations.

full rationale

This paper is an empirical challenge report, not a mathematical derivation. The 32.49% mAP on AVA and the 21.59% Kinetics-700 error rate are stated as measured results after training and testing on the respective benchmarks. They are not obtained by defining a quantity in terms of another quantity, nor by fitting a parameter and then relabeling the fit as a prediction. The proposed three-branch fusion is described qualitatively, and the paper does not present an equation that reduces a reported result to an input. Self-citations appear only as future work references: HAKE [1] is mentioned in the abstract and conclusion as a direction for future human activity knowledge, and RMPE [22] is mentioned as a possible future key-point feature source. Neither citation is load-bearing for the reported benchmark numbers. The most substantial weakness is that Table 2 compares methods with different backbones and pretraining datasets, so the causal attribution of the mAP gain to the three-branch design is not rigorously established by a controlled ablation. However, this is a validity or attribution concern, not circularity. Circularity would require the reported result to be equivalent to its inputs by construction, which is not the case here. The paper therefore receives a score of 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No new particles, forces, dimensions, or entities are introduced. The only new components are architectural branches built from existing modules. The free parameters are standard engineering choices, but several are not reported with enough detail to reproduce or justify.

free parameters (5)

Long-term feature bank size = 305 person features (5 persons per second, 61 seconds)
Chosen in Section 3 without an ablation; the paper states this selection but gives no sensitivity analysis.
Number of LFB blocks = 2
Section 3 says more than two blocks gives similar results, but no performance numbers are provided.
Human detection confidence threshold = 0.8
Section 4 uses detected boxes with confidence above 0.8 for testing; no analysis of this threshold is given.
Global loss weight = not specified
The global scene classifier is added without a stated weight relative to the per-person loss, affecting the fusion balance.
Focal loss hyperparameters = not specified
Focal loss is adopted, but alpha and gamma values are not reported.

assumptions (3)

domain assumption Features learned on Kinetics-600 and Kinetics-700 transfer to AVA action localization.
The backbone is pretrained on Kinetics and then fine-tuned on AVA; the paper assumes this transfer is a valid basis for the final classifier (Section 4).
domain assumption Human detections with confidence above 0.8 are accurate enough for action classification.
Test-time boxes come from Faster R-CNN with a single confidence cutoff, and no detection failure analysis is provided (Section 4).
domain assumption The AVA test labels and evaluation protocol used by the challenge are the ground truth for the reported mAP.
The 32.49% mAP is self-reported against an external benchmark, but no official evaluation output is included (Section 4, Table 2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Three Branches: Detecting Actions With Richer Features." pith.science (2026). https://pith.science/paper/XX27XQQH

@misc{pith2026190804519,
  author       = {Pith},
  title        = {Pith review of: Three Branches: Detecting Actions With Richer Features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XX27XQQH}},
  note         = {Machine review of arXiv:1908.04519}
}

read the original abstract

We present our three branch solutions for International Challenge on Activity Recognition at CVPR2019. This model seeks to fuse richer information of global video clip, short human attention and long-term human activity into a unified model. We have participated in two tasks: Task A, the Kinetics challenge and Task B, spatio-temporal action localization challenge. For Kinetics, we achieve 21.59% error rate. For the AVA challenge, our final model obtains 32.49% mAP on the test sets, which outperforms all submissions to the AVA challenge at CVPR 2018 for more than 10% mAP. As the future work, we will introduce human activity knowledge, which is a new dataset including key information of human activity.

Figures

Figures reproduced from arXiv: 1908.04519 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 7 canonical work pages

[1]

HAKE: human activity knowledge engine

Yonglu Li, Liang Xu, Xijie Huang, Xinpeng Liu, Ze Ma, Mingyang Chen, Shiyi Wang, Haoshu Fang, and Cewu Lu. HAKE: human activity knowledge engine. CoRR, abs/1904.06539, 2019

arXiv 1904
[2]

Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri

Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. C3D: generic features for video analysis. CoRR, abs/1412.0767, 2014

arXiv 2014
[3]

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. CoRR, abs/1711.10305, 2017

work page Pith review arXiv 2017
[4]

Rethinking spatiotemporal feature learning for video understanding

Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. CoRR, abs/1712.04851, 2017. 4

arXiv 2017
[5]

Quo vadis, action recognition? A new model and the kinetics dataset

João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CoRR, abs/1705.07750, 2017

arXiv 2017
[6]

Slowfast networks for video recognition

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. CoRR, abs/1812.03982, 2018

arXiv 2018
[7]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017

arXiv 2017
[8]

Girshick, Abhinav Gupta, and Kaiming He

Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CoRR, abs/1711.07971, 2017

arXiv 2017

Show all 22 references

[9]

Video action transformer network

Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. CoRR, abs/1812.02707, 2018

2018 arXiv
[10]

Girshick

Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, and Ross B. Girshick. Long-term feature banks for detailed video understanding. CoRR, abs/1812.05038, 2018

2018 arXiv
[11]

The kinetics human action video dataset

Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017

2017 arXiv
[12]

A short note about kinetics-600

João Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. CoRR, abs/1808.01340, 2018

2018 arXiv
[13]

Moments in time dataset: one million videos for event understanding

Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl V ondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Inte...

2019
[14]

F. C. Heilbron, V . Escorcia, B. Ghanem, and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 961–970, June 2015

2015
[15]

Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta

Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, 2016

2016
[16]

Ross, George Toderici, Yeqing Li, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik

Chunhui Gu, Chen Sun, Sudheendra Vijayanarasimhan, Caroline Pantofaru, David A. Ross, George Toderici, Yeqing Li, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. A V A: A video dataset of spatio-temporally localized atomic visual actions. CoRR, abs/1705.0...

2017 arXiv
[17]

Girshick, and Jian Sun

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, abs/1506.01497, 2015

2015 arXiv
[18]

Girshick, Kaiming He, and Piotr Dollár

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. CoRR, abs/1708.02002, 2017

2017 arXiv
[19]

YH technologies at activitynet challenge 2018

Ting Yao and Xue Li. YH technologies at activitynet challenge 2018. CoRR, abs/1807.00686, 2018

2018 arXiv
[20]

A better baseline for A V A

Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. A better baseline for A V A. CoRR, abs/1807.10066, 2018

2018 arXiv
[21]

Belongie, Lubomir D

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014

2014 arXiv
[22]

RMPE: regional multi-person pose estimation

Haoshu Fang, Shuqin Xie, and Cewu Lu. RMPE: regional multi-person pose estimation. CoRR, abs/1612.00137, 2016. 5

2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

HAKE: human activity knowledge engine

Yonglu Li, Liang Xu, Xijie Huang, Xinpeng Liu, Ze Ma, Mingyang Chen, Shiyi Wang, Haoshu Fang, and Cewu Lu. HAKE: human activity knowledge engine. CoRR, abs/1904.06539, 2019

arXiv 1904

[2] [2]

Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri

Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. C3D: generic features for video analysis. CoRR, abs/1412.0767, 2014

arXiv 2014

[3] [3]

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. CoRR, abs/1711.10305, 2017

work page Pith review arXiv 2017

[4] [4]

Rethinking spatiotemporal feature learning for video understanding

Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. CoRR, abs/1712.04851, 2017. 4

arXiv 2017

[5] [5]

Quo vadis, action recognition? A new model and the kinetics dataset

João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CoRR, abs/1705.07750, 2017

arXiv 2017

[6] [6]

Slowfast networks for video recognition

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. CoRR, abs/1812.03982, 2018

arXiv 2018

[7] [7]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017

arXiv 2017

[8] [8]

Girshick, Abhinav Gupta, and Kaiming He

Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CoRR, abs/1711.07971, 2017

arXiv 2017