REVIEW 3 major objections 4 minor 33 references
ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition
T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read ALBAR claims that training a video encoder against motionless clips—one repeated frame—suppresses both background and foreground static bias without attribute labels, lifting HMDB51 contrasted accuracy from 40.28% to 53.22%.
desk verdict The method is clever and the benchmark fix is useful, but the headline 12-point gain comes from a metric that is literally the training objective, not from demonstrated foreground debiasing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the static clip $\bar{x}^{(i)}_{\bar{t}}$: one frame sampled from the input clip and replicated so that it contains zero motion. Passing it through the same encoder $F$ as the moving clip creates a paired-input contrast where the only difference between the two inputs is temporal dynamics. The adversarial cross-entropy loss reverses the static clip's gradient, the entropy-maximization loss forces its class predictions toward the uniform distribution, and the gradient-norm penalty keeps the encoder from reacting violently to static inputs; together they make the encoder dependent on motion while remaining ignorant of any static pixel pattern, with no knowledge of what the biased attribute is.
What would settle it
Run ALBAR on an action set where the class is defined by a static attribute—say, distinguishing 'playing guitar' from 'playing violin' from the instrument held in a single frame. The adversarial and entropy objectives predict near-uniform predictions on the motionless SCUFO clips; if the trained model cannot classify such single frames above chance while an ordinary ERM baseline can, then ALBAR's low SCUFO accuracy reflects discarded static signal rather than elimination of spurious correlation. A second check is to break down the 1.50% HMDB51 SCUFO accuracy per class and ask whether the near-zero scores are uniform or concentrated in classes whose action is visually defined by a static object.
Extended reading notes
Core claim
The central claim is that static spatial information—background scene cues and foreground appearance cues alike—can be neutralized by feeding the model an adversarial motionless version of its input. For a clip $x^{(i)}_t$, ALBAR builds a static clip $\bar{x}^{(i)}_{\bar{t}}$ by repeating one sampled frame, passes both clips through the same encoder $F$, and trains with a cross-entropy loss on the moving clip while negating the cross-entropy gradient for the static clip (Eq. 2). Entropy maximization (Eq. 3) pushes the static-clip prediction toward a uniform distribution, avoiding the trivial solution where the model confidently predicts a wrong class from static cues, and a gradient penalty (Eq. 4) stabilizes training by limiting the encoder's sensitivity to static input. On HMDB51, this yields 53.22% contrasted accuracy versus 40.28% for StillMix, and it transfers to Kinetics400 and UCF101; the same recipe improves downstream anomaly detection and temporal action localization.
Load-bearing premise
The paper's own limitation states the load-bearing premise: no static information is ever useful for recognizing an action. If some actions do require static cues—such as 'playing guitar' or 'reading'—forcing uniform predictions on static clips removes signal rather than bias, and the headline 12-point gain may partly reflect that removal rather than true debiasing.
Editorial extensions
If this is right
- HMDB51 SCUBA/SCUFO contrasted accuracy reaches 53.22%, more than 12 absolute points above the previous best StillMix (40.28%), with IID accuracy within roughly one point of the ERM baseline.
- The method needs no bias-attribute labels and no separate critic network, so it can be dropped into any standard video encoder; combining it with StillMix augmentation pushes contrasted accuracy to 53.68% on HMDB51 and 31.14% on the repaired UCF101 protocol.
- Replacing the UCF101 protocol's bounding boxes with tight segmentation masks removes a background-leakage shortcut, giving a more faithful measurement of debiasing capability, and ALBAR also leads on that repaired benchmark.
- Debiasing transfers to downstream video tasks: with unchanged hyperparameters, ALBAR features improve UCF_Crime anomaly detection AUC from 82.39 to 84.91 and THUMOS14 action localization mAP from 54.89 to 55.20.
Reading between the lines
- The paired-input recipe is general: any domain where a degraded input can be produced that shares most information with the original but lacks the discriminative signal (e.g., spatial downsampling, background removal, or masked objects) could use the same single-encoder adversarial objective to strip shortcut information.
- The paper's assumption that no static information is useful could be tested directly by measuring per-class SCUFO accuracy: classes with genuinely diagnostic static cues should show non-uniform errors if the model is truly debiasing, versus uniform randomness if it is merely maximizing entropy.
- The improved UCF101 protocol depends on the quality of the segmentation model (SAMTrack and Segment Anything); as segmentation improves or fails on difficult actors, the benchmark numbers will shift, so comparing methods on it should control for the exact mask generation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ALBAR, an adversarial training method for video action recognition that uses a single 3D encoder with three loss components: a negative-gradient adversarial cross-entropy loss on static clips (a single frame repeated to form a motionless clip), an entropy-maximization loss that drives static-clip predictions toward a uniform distribution, and a gradient-penalty regularizer for training stability. The method is evaluated on the SCUBA/SCUFO background/foreground bias protocols for HMDB51, UCF101, and Kinetics400, reporting a contrasted accuracy of 53.22% on HMDB51, an absolute gain of roughly 12.94% over the previous best method StillMix (40.28%). The paper also identifies background leakage in the existing UCF101 SCUBA/SCUFO protocol and proposes a refined version using tight segmentation masks from SAMTrack, on which ALBAR also outperforms prior methods. Additional experiments on ARAS rare-scene evaluation and downstream tasks (anomaly detection, temporal action localization) are reported, along with ablations of the loss components and static-frame sampling strategy.
Significance. If the claims hold, ALBAR is a noteworthy contribution: it achieves strong debiasing performance without attribute labels or a separate critic model, using a simple single-encoder adversarial objective. The ablation study is thorough and shows that each loss component contributes to the final result. The identification of background leakage in the UCF101 protocol and the proposed segmentation-based fix address a real limitation of an established benchmark. The downstream task evaluations provide some evidence that the debiased encoder retains useful temporal features. However, the headline foreground-debiasing result rests on the SCUFO metric, which is directly optimized by the training objective, so the claimed 12-point improvement on contrasted accuracy is substantially a measure of how well the model satisfies its own loss rather than independent evidence of reduced foreground bias.
major comments (3)
- [§3.2, §3.3, Table 1] The training objective in Eqs. (2) and (3) directly operates on static clips (a single frame repeated) by maximizing cross-entropy and maximizing entropy of the predictions, which drives the model to output uniform predictions on exactly the same kind of input used in SCUFO evaluation. The reported SCUFO accuracy (1.50% on HMDB51) is therefore a direct reflection of training-objective satisfaction, not an independent measurement of reduced foreground bias. The contrasted-accuracy gain over StillMix (53.22 vs 40.28) is almost entirely attributable to the SCUFO drop (from 13.39% to 1.50%); SCUBA improves by only 1.72 points. To support the claim that the method genuinely mitigates foreground bias rather than merely satisfying the loss, please provide additional evidence, such as per-class SCUFO accuracy (especially for actions with legitimate static cues like 'playing guitar' or 'reading'), or a probing analysis showing that the learned representations are invariant to foreground appearance changes.
- [Limitations (Section 5 in main text, after Table 5)] The Limitations section concedes that the approach 'makes an assumption that no static information should be useful.' This assumption is load-bearing: for actions where static cues are informative (e.g., smoking, playing guitar, reading), forcing uniform predictions on static clips removes usable signal, so low SCUFO accuracy constitutes over-debiasing rather than bias removal. The paper does not quantify how the method affects accuracy on such classes, either on the IID test set or in a per-class SCUFO analysis. Please add a discussion or experiment that addresses the trade-off between removing spurious static correlations and discarding legitimate static information, otherwise the headline gain is presented as pure debiasing when part of it may be harmful signal suppression.
- [Table 8 and §4.2 and Table 2] On the existing UCF101 SCUBA/SCUFO protocol, ALBAR achieves a contrasted accuracy of 47.88%, well below StillMix's 54.90% (Table 8). The authors argue this is due to background leakage in the old protocol, but on their new protocol the margin over StillMix is only 1.83 points (26.23 vs 24.40 in Table 2). This is a much smaller advantage than the 12.94-point gain on HMDB51. The paper should make explicit that the SOTA claim on UCF101 is only on the authors' proposed protocol, not on the previously established one, and should provide a deeper analysis of how the two protocols differ (e.g., agreement between old and new SCUFO/SCUBA sets). Additionally, since the new masks were created with manual inspection, the paper should describe the criteria used for manual checking and make the masks and protocol-generation code publicly available to allow independent verification.
minor comments (4)
- [§4.4, text near Table 1] The text says 'on the SCUFA protocol' but the benchmark is named SCUFO; please correct the typo.
- [§3.2] The static clip is created by stacking a sampled frame F times; please state explicitly whether F equals the number of frames in the motion clip (presumably 32) and whether the same random augmentations are applied to the static clip as to the motion clip.
- [§4.5, Table 5] The downstream task gains on UCF_Crime (82.39 to 84.91 AUC) and THUMOS14 (54.89 to 55.20 mAP) are reported without standard deviations or significance tests; please provide run-to-run variance or at least acknowledge that these differences may be within noise.
- [Appendix, Waterbirds experiment] The Waterbirds generalization experiment is a nice addition, but the adaptation of the method to image classification (removing the foreground) should be described in more detail, and reporting a single run per setting is insufficient; consider reporting mean and std over multiple seeds.
Circularity Check
Headline contrasted-accuracy gain is partly the training objective itself: SCUFO clips are the same repeated-frame static clips that Eqs. 2-3 force toward incorrect/uniform predictions, so the 12-point gain is not independent evidence of foreground debiasing.
-
fitted input called prediction
[Sec. 3.2 (static clip construction, Eq. 2), Sec. 3.3 (Eq. 3), Sec. 4.1 (SCUFO definition), Sec. 4.4 (contrasted accuracy definition)]
"we sample a single frame at a specified time ¯t within x(i)t, stacking it F times to create static clip, ¯x(i)¯t ... the cross-entropy between the prediction vector of the static clip ¯p(i)¯t and y(i) is maximized ... By encouraging the model to predict all classes with an equal probability given a clip with no motion ... SCUFO takes a frame from a SCUBA video and stacks it into a motionless clip to evaluate potential foreground bias ... a video prediction is counted correct if and only if the model is correct on a SCUBA video and is incorrect on the corresponding SCUFO video."
The static clip used in training is constructed by repeating one sampled frame, and the SCUFO evaluation clip is constructed the same way: a frame from a SCUBA video stacked into a motionless clip. Equations (2) and (3) explicitly train the model to maximize cross-entropy on such clips and to drive their softmax outputs toward uniform, i.e., to be incorrect and uncertain on exactly the inputs used to measure SCUFO. Contrasted accuracy then rewards incorrect SCUFO predictions by construction. On HMDB51, ALBAR's SCUBA accuracy is only 1.72 points above StillMix (53.53 vs. 51.81), while its SCUFO accuracy is 11.89 points lower (1.50 vs.
full rationale
ALBAR's training objective and the SCUFO half of the headline metric are the same operation. Section 3.2-3.3 builds a static clip by repeating one sampled frame and applies L_adv = -CE(static prediction, label) plus entropy maximization to drive the prediction toward uniform. Section 4.1 defines SCUFO by taking a frame from a SCUBA video and stacking it into a motionless clip; Section 4.4 defines contrasted accuracy as correct on SCUBA and incorrect on the paired SCUFO clip. Thus the model is explicitly optimized to be wrong and uncertain on exactly the clips used to measure foreground bias. The 12.94-point HMDB51 contrasted-accuracy gain versus StillMix decomposes into a 1.72-point SCUBA gain and an 11.89-point SCUFO drop, so the gain is dominated by the directly optimized term. The Limitations section concedes that treating all static information as useless may discard legitimate cues such as playing guitar or reading; low SCUFO accuracy is therefore over-debiasing as much as debiasing. Independent evidence (SCUBA, Confl-FG, ARAS, downstream tasks, and the new UCF protocol) supports some genuine improvement in background robustness and makes the paper only partially circular. No load-bearing self-citation chain or imported uniqueness theorem is present. Score 6 reflects one central prediction (contrasted accuracy via SCUFO) reducing by construction to the training objective.
Assumptions & free parameters
free parameters (4)
- omega_adv =
1
- omega_ent =
4
- omega_gp =
10
- static frame sampling strategy =
middle
assumptions (3)
- ad hoc to paper All static (single-frame) information is spurious for action recognition
- domain assumption SCUBA/SCUFO contrasted accuracy is a valid measure of background-plus-foreground debiasing
- domain assumption A single repeated frame is a faithful proxy for all static spatial cues
Cite this review
Pith. "Pith review of ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition." pith.science (2026). https://pith.science/paper/UJIQI722
@misc{pith2026250200156,
author = {Pith},
title = {Pith review of: ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/UJIQI722}},
note = {Machine review of arXiv:2502.00156}
}
read the original abstract
Bias in machine learning models can lead to unfair decision making, and while it has been well-studied in the image and text domains, it remains underexplored in action recognition. Action recognition models often suffer from background bias (i.e., inferring actions based on background cues) and foreground bias (i.e., relying on subject appearance), which can be detrimental to real-life applications such as autonomous vehicles or assisted living monitoring. While prior approaches have mainly focused on mitigating background bias using specialized augmentations, we thoroughly study both foreground and background bias. We propose ALBAR, a novel adversarial training method that mitigates foreground and background biases without requiring specialized knowledge of the bias attributes. Our framework applies an adversarial cross-entropy loss to the sampled static clip (where all the frames are the same) and aims to make its class probabilities uniform using a proposed entropy maximization loss. Additionally, we introduce a gradient penalty loss for regularization against the debiasing process. We evaluate our method on established background and foreground bias protocols, setting a new state-of-the-art and strongly improving combined debiasing performance by over 12% absolute on HMDB51. Furthermore, we identify an issue of background leakage in the existing UCF101 protocol for bias evaluation which provides a shortcut to predict actions and does not provide an accurate measure of the debiasing capability of a model. We address this issue by proposing more fine-grained segmentation boundaries for the actor, where our method also outperforms existing approaches. Project Page: https://joefioresi718.github.io/ALBAR_webpage/
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
train/test split for all training and evaluation in this work. HMDB51 Kuehne et al. (2011) consists of 6,849 video clips covering 51 human activity classes and has three potential train/test splits, much like UCF101. Again, we only use the first (split
work page 2011
-
[5]
Shortcut learning in deep neural networks
11 Published as a conference paper at ICLR 2025 Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673,
work page 2025
-
[6]
Similar outcomes to the other experiments are seen here. 16 Published as a conference paper at ICLR 2025 Table 6: Results on IID and OOD test sets of various debiasing methods on Kinetics400. All experiments use Swin-T pretrained using Kinetics-400. Augmentation or Debiasing IID OOD Avg SCUBA x Avg SCUFO y ARAS x Contra. Acc. x None 68.13 42.97 20.26 ...
work page 2011
-
[7]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
-
[8]
Segment anything.arXiv preprint arXiv:2304.02643,
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything.arXiv preprint arXiv:2304.02643,
-
[9]
Hmdb: a large video database for human motion recognition
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pp. 2556–2563. IEEE,
work page 2011
-
[10]
Grounding dino: Marrying dino with grounded pre-training for open-set object detection
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499,
-
[11]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,
Show all 33 references
-
[14]
Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731,
1911 arXiv
-
[15]
When does bias transfer in transfer learning? arXiv preprint arXiv:2207.02842,
Hadi Salman, Saachi Jain, Andrew Ilyas, Logan Engstrom, Eric Wong, and Aleksander Madry. When does bias transfer in transfer learning? arXiv preprint arXiv:2207.02842,
-
[16]
On the integration of optical flow and action recognition
Laura Sevilla-Lara, Yiyi Liao, Fatma Güney, Varun Jampani, Andreas Geiger, and Michael J Black. On the integration of optical flow and action recognition. In Pattern Recognition: 40th German Conference, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings 40, pp. 281...
2018
-
[17]
Ucf101: A dataset of 101 human actions classes from videos in the wild
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,
-
[19]
Convnets and imagenet beyond accuracy: Explanations, bias detection, adversarial examples and model criticism
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Explanations, bias detection, adversarial examples and model criticism. arXiv preprint arXiv:1711.11443,
-
[21]
Investigating bias in image classification using model explanations
Schrasing Tong and Lalana Kagal. Investigating bias in image classification using model explanations. arXiv preprint arXiv:2012.05463,
2012 arXiv
-
[22]
The caltech-ucsd birds-200-2011 dataset
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset
2011
-
[23]
Mimetics: Towards understanding human actions out of context
13 Published as a conference paper at ICLR 2025 Philippe Weinzaepfel and Grégory Rogez. Mimetics: Towards understanding human actions out of context. International Journal of Computer Vision, 129(5):1675–1690,
2025
-
[24]
Predictive inequity in object detection
Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive inequity in object detection. arXiv preprint arXiv:1902.11097,
1902 arXiv
-
[25]
Videomix: Rethinking data augmentation for video classification
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457,
2012 arXiv
-
[26]
Mitigating unwanted biases with adversarial learning
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335– 340,
2018
-
[27]
mixup: Beyond empirical risk minimization
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412,
-
[28]
Men also like shopping: Reducing gender bias amplification using corpus-level constraints
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457,
-
[29]
It has a single dedicated train/val/test split
14 Published as a conference paper at ICLR 2025 APPENDIX OVERVIEW Section A: Dataset details Section B: Implementation/compute details Section C: Additional experiment details A D ATASET DETAILS Kinetics400 Carreira & Zisserman (2017) contains approximately 300,000 video clips...
2017
-
[31]
ARAS Duan et al
train/test split for all training and evaluation in this work. ARAS Duan et al. (2022) is only a test set, so all of the 1,038 rare-scene action videos are utilized to compute Top-1 accuracy. SCUBA and SCUFO Li et al. (2023) are also test sets. Each dataset variation contains ...
2022
-
[32]
Golf Swing
Note that in the “Fencing” example, we additionally segment the opposing fencer for a more complete video. 15 Published as a conference paper at ICLR 2025 (a) (b) (c) (d) Figure 5: Example clips from UCF101-SCUBA-Places365 and UCF101-SCUBA-VQGAN protocols. (a) shows an example...
2019
-
[2012]
Image representations learned with unsupervised pre-training contain human-like biases
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 701–713,
2021
-
[2014]
Accurate, large minibatch sgd: Training imagenet in 1 hour
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677,
-
[2017]
A framework for understanding unintended consequences of machine learning
Harini Suresh and John V Guttag. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002, 2(8),
1901 arXiv
-
[2018]
Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231,
-
[2019]
Large datasets: A pyrrhic win for computer vision
12 Published as a conference paper at ICLR 2025 Vinay Uday Prabhu and Abeba Birhane. Large datasets: A pyrrhic win for computer vision. arXiv preprint arXiv:2006.16923, 3,
2025 arXiv
-
[2020]
arXiv preprint arXiv:2001.09762,
2001 arXiv
-
[2021]
Data decisions and theoretical implications when adversarially learning fair representations
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00075,
-
[2022]
Adversarial removal of demographic attributes from text data
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. arXiv preprint arXiv:1808.06640,
-
[2023]
Segment and track anything
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06558,
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.