Cross-Modal Dual-Causal Learning for Long-Term Action Recognition

Chang Jing; Fan Chao; Gao Junyu; Jia Xibin; Sun Qianmei; Xu Shaowu

REVIEW 5 major objections 5 minor 51 references

Cross-Modal Dual-Causal Learning for Long-Term Action Recognition

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dual-causal structural model strips cross-modal bias and visual confounders from vision-language based long-term action recognition, reporting state-of-the-art accuracy on Breakfast, COIN, and Charades.

desk verdict The dual-causal setup is new and the numbers are strong, but the front-door adjustment is invalid because the mediator is built from the confounded visual features, so the causal claim doesn't stand. read the letter →

arxiv 2507.06603 v2 pith:44XQNZFD submitted 2025-07-09 cs.CV

Xu Shaowu , Jia Xibin , Gao Junyu , Sun Qianmei , Chang Jing , Fan Chao This is my paper

classification cs.CV

keywords long-termactionrecognitioncausallearningstructuralmodelcross-modalbiasvisualconfoundersvision-languagemodelsback-dooradjustmentfront-door

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-term action recognition demands modeling how atomic actions relate in time while ignoring environmental clutter. Vision-language models (VLMs) are strong at matching video frames to text, but they inherit statistical biases: they over-associate visually prominent yet noncausal actions with the wrong long-term activity, and they confuse visually similar actions that appear in different activities. This paper proposes CMDCL, a structural causal model with two interventions: a textual back-door adjustment that debiases text embeddings, and a visual front-door adjustment, guided by the debiased text, that removes visual confounders such as background and attire. On Breakfast, COIN, and Charades, CMDCL reports state-of-the-art accuracy and mAP, and the authors argue the gains come from deconfounding rather than from more parameters. The approach, however, hinges on a key assumption: the debiased text embedding that acts as mediator is itself built from the video's visual features, so its independence from the visual confounders is not guaranteed.

What carries the argument

The load-bearing object is a structural causal model with five latent variables: video feature $V$, text feature $T$, label $Y$, cross-modal bias $B$, and visual confounder $Z$, plus a mediator $M$. The paper defines the mediator as the debiased text embedding $M=T'$, obtained from the textual causal intervention. The text intervention uses back-door adjustment, estimating $P(B)$ via a learnable approximator and fusing the visual prompt with text via the Normalized Weighted Geometric Mean approximation. The visual intervention uses front-door adjustment: it assumes $T'$ is independent of $Z$, then computes deconfounded visual embeddings by attending to video features with scores measured against $T'$. The two interventions together are intended to cut the back-door paths $\{V,T\} \leftarrow B \rightarrow Y$ and $V \leftarrow Z \rightarrow Y$.

What would settle it

Take one action sequence, such as 'make scrambled egg', and collect or render it with varied backgrounds, lighting, and actor attire while keeping the hand-object interactions identical; if CMDCL's predicted label or its deconfounded visual embedding changes substantially across these variants, then the visual confounders still reach the mediator and the front-door adjustment's key independence assumption is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that both cross-modal bias and visual confounders can be removed from VLM-based long-term action recognition by applying two explicit causal interventions. The first, textual causal intervention, performs a back-door adjustment on the text feature $T$, approximating $P(Y|V,do(T))$ by averaging over a surrogate for the bias $B$ derived from visual prompts; this produces a debiased text embedding $T'$. The second, visual causal intervention, defines a mediator $M=T'$ and applies a front-door adjustment to the video feature $V$, using spatio-temporal attention and an approximator to get a deconfounded visual embedding $V'$. The model then classifies from the pair $(T',V')$. The authors report that this dual deconfounding yields state-of-the-art numbers on Breakfast (96.62% Acc, 89.10% mAP), COIN (92.28% Acc@1, 99.28% Acc@5, 85.51% mAP), and Charades (51.1% mAP), and they provide attention-map and matching-probability analyses showing that the model concentrates on hand-object interactions rather than background and attire.

Load-bearing premise

The mediator is the debiased text embedding, and the front-door adjustment assumes this embedding is independent of the visual confounders, yet the embedding is computed from the video's own visual features, so it may carry those confounders with it.

Editorial extensions

If this is right

If the dual causal interventions work as described, VLM-based long-term action recognition can achieve strong accuracy without pre-trimmed single-label videos, as the Charades result (51.1% mAP on full multi-label videos) shows.
The success on Breakfast and COIN suggests that explicit causal adjustments can be layered on top of a frozen pretrained VLM, making deconfounding a viable upgrade for existing VLM classifiers.
The debiased text embeddings $T'$ from the textual intervention could be reused as a drop-in replacement for raw text embeddings in other vision-language tasks where cross-modal bias is suspected.
The independent video encoder in the visual intervention provides a concrete way to block back-door information inherited from the pretrained VLM, which the paper identifies as a source of spurious correlation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own construction strains the mediator assumption: $T'$ is computed from the video's visual features, so the mediator $M$ may inherit the visual confounders it is supposed to block, and the front-door adjustment may not fully remove the $Z$ back-door path.
A direct test of the independence assumption is to recompute $T'$ for the same action under different backgrounds or lighting; if $T'$ shifts, the mediator is confounded and the causal story weakens to a conditional-model claim.
The Charades comparison mixes training regimes: pre-trimmed methods such as TokenLearner reach 66.3% mAP, while CMDCL uses full multi-label videos; a fair ablation would train a pre-trimmed variant of CMDCL to isolate the causal modules' true contribution.
A natural extension is to replace the mediator with an explicitly disentangled representation trained to be invariant to visual confounders (e.g., via adversarial or domain-robust objectives), which would make the front-door assumption testable and potentially more robust.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This one is worth a look, but the headline causal claim doesn't survive contact with the paper. The combination of back-door adjustment on text and front-door adjustment on video for VLM-based long-term action recognition is new, and the reported numbers are striking: 96.62% Acc and 89.10% mAP on Breakfast, 92.28/99.28 on COIN, and 51.1% mAP on Charades, beating BIKE and MA-LMM. Code is promised.

What the paper does well: it identifies two real failure modes in VLM-based LTAR (cross-modal bias and visual confounders), and the proposed TCI and VCI modules are concrete, with the ablation showing both contribute. The idea of using debiased text as a mediator is clever, and the independent video encoder is a sensible way to block the VLM's biased visual features. As an architecture paper, it's a reasonable contribution.

The problem is the causal identification. The mediator M is defined as the debiased text T' (Section 3.2.2), and T' is constructed in Eq. 3 from the VLM visual features V^P via the bias-aware scores. So M inherits the visual confounders Z through the path Z→V→M. The paper's own property 2 says M is independent of Z, but that's false by construction. Consequently Eq. 4, P(M|do(V))=P(M|V), is not an interventional equality, and the front-door estimand in Eq. 6 is not identified from the graph in Fig. 3. This is not a minor footnote; it's the load-bearing wall. The stress-test note gets this exactly right.

The other soft spots are proportionally less severe but still real. The derivation from Eq. 1 onwards is loose: Eq. 1 is not a valid factorization of P(Y|V,T) under the stated graph, and the do-calculus steps are asserted rather than derived. The NWGM approximations replace sums over discrete classes with concatenations and learned functions, so the gap between the causal formula and the implemented module is wide. Empirically, there are no error bars, and there's no non-causal attention baseline using the same components (e.g., the same STA layers and approximators without the intervention framing), so you can't attribute the gains to the causal machinery per se.

Who is this for? A reader working on causal debiasing of VLMs or on long-term action recognition, who wants architecture ideas and is willing to treat the causal language as motivation rather than proof. It deserves serious peer review because the empirical results are strong enough to check and the framework may be reusable. But the authors need to rewrite the causal sections, state the identification assumptions honestly, and add error bars plus a controlled baseline. As it stands, I would not rely on the causal claims.

Recommendation: send it to review, but the reviews should focus on the identification issue and the missing controls. For my own work, I wouldn't cite the causal result; I'd cite it only as an example of how not to set up a front-door adjustment.

Referee Report

5 major / 5 minor

Summary. The paper proposes Cross-Modal Dual-Causal Learning (CMDCL) for long-term action recognition. It introduces a structural causal model with a cross-modal bias B and a visual confounder Z, then claims to remove B via a textual back-door intervention (TCI) that produces debiased text embeddings T', and to remove Z via a visual front-door intervention (VCI) that uses T' as a mediator M to produce deconfounded visual embeddings V'. The method is evaluated on Breakfast, COIN, and Charades, with state-of-the-art accuracy/mAP claims, and the paper includes ablations, attention visualizations, and cross-modal matching statistics.

Significance. CMDCL addresses a relevant and under-explored problem: causal reasoning for VLM-based long-term action recognition. The paper is clearly written, ships code, and includes useful ablations and qualitative analyses. If the causal identification were sound, the dual-intervention framework would be a valuable contribution to the LTAR community. However, the identification argument is invalid: the debiased text embedding is constructed from the very visual features it is supposed to deconfound, and the front-door adjustment is applied to a graph that retains a direct V->Y path and contains a backdoor path between V and M. Consequently, the paper's central claim—that CMDCL performs dual causal interventions—is not supported by the derivations.

major comments (5)

[Section 3.1, Eq. (1)] The claimed Bayes expansion is algebraically invalid. P(Y|V,T) equals the sum over z and b of P(Y|V,T,z,b) P(z,b|V,T), but the paper replaces P(z,b|V,T) with P(z|V,T)P(b|T). This factorization requires conditional independencies that do not follow from the SCM in Figure 3; in particular, B and Z are both parents of V, so conditioning on V opens an M-gate that induces dependence between B and Z. Equation (1) is therefore not a valid starting point for the subsequent back-door adjustment.
[Section 3.2.1, Eq. (3)] The debiased text embedding T' is computed as sum_c h([t_c, b_c]), where b_c = sum_l s_{c,l} v^P_l and v^P_l are pretrained VLM visual features. Thus T' is a deterministic function of the observed video features, and any visual confounder Z that affects v^P propagates into T'. This means the textual intervention is not a do-operation on T; the implementation conditions on V. It also contradicts the later assertion in Section 3.2.2 that the mediator M=T' is independent of visual confounders because it is derived from the textual modality.
[Section 3.2.2, Eq. (4)] Setting P(M|do(V)) = P(M|V) requires the absence of backdoor paths between V and M. The paper asserts this because M is 'derived from text,' but M is in fact built from V^P and from T, while T is affected by B and V is also affected by B. Hence the path V <- B -> T -> M is a backdoor path between V and M. The independent video encoder blocks B->V only for the separately extracted V, not for the V^P used to construct M. No do-calculus steps are provided to justify Eq. (4).
[Section 3.2.2, Eqs. (5)-(6)] Front-door adjustment requires the mediator M to intercept all directed paths from V to Y. The paper's own SCM retains a direct V->Y path, described as the true causal effect V->Y<-T. With a direct path, Eq. (6) does not identify P(Y|do(V),T); at best it identifies a mediated component. The claim that VCI blocks Z->V->Y is therefore unsupported, because the confounding of the direct V->Y path by Z remains.
[Section 4, Tables 1-3] All reported results are single-run with no standard deviations, error bars, or significance tests. Several headline comparisons are very close (e.g., 96.62 vs 95.77 on Breakfast Acc@1; 92.28 vs 93.20 on COIN Acc@1; 51.1 vs 50.4 on Charades mAP). Given that the causal claim is the main differentiator, the empirical evidence does not independently corroborate the causal mechanism; repeated runs and significance testing are needed.

minor comments (5)

[Section 3.2.1, Eq. (3)] The NWGM notation P(Y|V, sum_c concat(...)) is informal; the conditioning event contains a sum over class indices and should be defined as a feature-level approximation rather than a probability conditional on a vector expression.
[Section 3.2.2, Eq. (7)] The expression concat(M=t'_c, V=v_c,) contains a stray comma, and the densities P(V=v_c) and P(M=t'_c|V) are not explicitly defined as discrete or continuous.
[Table 2] The pre-trimmed versus untrimmed training protocol is a major difference across methods; the caption should state clearly that CMDCL is trained on untrimmed multi-label videos and that comparisons with pre-trimmed methods should be interpreted accordingly.
[References] References [20] and [21] are the same paper (Liu, Li, and Lin, TPAMI 2023) but are listed and cited as separate entries; please unify them.
[Figure 7] The caption contains repeated typos: 'Effection' should be 'Effect' in both subfigure captions.

Circularity Check

2 steps flagged · score 8.0 of 10

Mediator M=T' is built from confounded visual features (Eq. 3), so the front-door deconfounding in Eqs. 4-6 is circular by construction; fitted approximators relabel causal estimates.

self definitional [Section 3.2.1 Eq. (3); Section 3.2.2 property 2 and Eq. (4)]
"TCI fuses scores S with V^P to obtain bias-aware embeddings B={b_c | \sum_{l \in L} s_{c,l}v^P_l} ... \approx P(Y|V,\sum_{c=1}^C h([t_c,b_c]))=P(Y|V,T'), (3) ... Thus, VCI introduces the mediator M, defined as M=T', ... As M is derived from textual modality and independent of visual confounders Z, there are no back-door paths between V and M involving Z."

Eq. (3) defines T' as a learned function of b_c, and b_c is the weighted sum of VLM visual features V^P. In the paper's own SCM (Fig. 3), visual features are descendants of the visual confounder Z (Z→V), so T' inherits Z through Z→V^P→B→T'. Yet property 2 in Sec. 3.2.2 asserts M=T' is 'derived from textual modality and independent of visual confounders Z.' That independence is therefore imposed by definition rather than established by construction. Since Eq. (4) then equates P(M|do(V)) with the observational P(M|V), the front-door adjustment in Eqs. (5)-(6) operates on a confounded distribution and cannot block Y←Z→V→M. The claimed deconfounding reduces to definitional assumption, not causal identification.
fitted input called prediction [Section 3.2.1 Eq. (3); Section 3.2.2 Eq. (8); Section 4.1 training protocol]
"Finally, VCI employs a learnable approximator g(\cdot) to approximate P(M=t'_c|V) and P(V=\hat{v}_c). This produces deconfounded visual embeddings V' = {v'_c}, enabling: P(Y|do(V),T) \approx P(Y| \sum_c g([t'_c,\hat{v}_c]),V)=P(Y|V',T). (8) Models are trained for 80 epochs with a cosine decay scheduler and an initial learning rate of 1e-5 using the Adam optimizer."

The 'debiased' text T' and 'deconfounded' visual embeddings V' are not fixed interventional quantities: T' = \sum_c h([t_c,b_c]) with h a learnable approximator, and V' = \sum_c g([t'_c,\hat{v}_c]) with g a learnable approximator. These parameters are trained end-to-end on the action-recognition labels under the Sec. 4.1 protocol, so the causal-looking outputs are fitted functions of the same inputs the model is supposed to deconfound. Presenting them as P(Y|do(T),V) and P(Y|do(V),T) relabels fitted features as causal estimates; the 'prediction' is forced by the learned transforms rather than by an independent intervention.

full rationale

The central causal claim—that VCI removes visual confounders via front-door adjustment—is circular. Section 3.2.2 defines the mediator as M=T' and justifies front-door adjustment on the assertion that M is 'derived from textual modality and independent of visual confounders Z.' But Eq. 3 constructs T' by h([t_c,b_c]) with b_c=\sum_l s_{c,l}v^P_l, a function of pretrained VLM visual features that are themselves descendants of Z in the paper's SCM. Therefore the required independence of the mediator from Z is asserted, not satisfied, and Eq. 4's P(M|do(V))=P(M|V) is the observational, confounded distribution. Separately, both T' and the final deconfounded V' are produced by learnable approximators h,g trained end-to-end on the action labels, so the 'interventional' embeddings are fitted quantities relabeled as causal estimates. This is not a case of a few benign self-citations; the load-bearing identification step reduces to a definition that Eq. 3 contradicts. The empirical gains may still come from the architecture, but the paper's causal deconfounding derivation is not self-contained.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

The central causal claims rest on latent variables B and Z that are not observed, a mediator that is defined from the confounded video features, and two learned approximators trained end-to-end on the target data. There is no external benchmark for the causal quantities, so the entire load rests on unverifiable assumptions and fitted parameters.

free parameters (5)

Learnable text embeddings T (initialized from CLIP) = not reported
Finetuned per benchmark; the debiased text embeddings T' depend on these values.
Approximator networks h and g = not reported
Trained end-to-end; their outputs define T' and V', which are the core of the claimed interventions.
Classifier weights w_c,d and bias b_c = not reported
Standard linear classifier trained on each dataset.
Number of STA layers H = 6
Chosen based on the hyperparameter study in Figure 7a; 8 layers give negligible improvement, so 6 is used.
Softmax temperature tau = not reported
Used in the attention scoring formulas (Eqs. 2 and VCI scoring) but not specified in the paper.

assumptions (4)

ad hoc to paper The SCM in Figure 3 correctly represents the causal structure of VLM-based LTAR, with separable cross-modal bias B and visual confounders Z.
The graph is assumed, not learned or validated, and the unobservable B and Z are operationalized through attention-weighted aggregations.
domain assumption The Normalized Weighted Geometric Mean (NWGM) yields a valid approximation of the interventional distributions in Eqs. 3 and 7.
NWGM is borrowed from attention literature, but its application to sums over unobserved confounders is not justified.
ad hoc to paper The mediator M = T' is independent of the visual confounders Z.
Stated as property 2 in Section 3.2.2, but T' is computed from visual features (Eq. 3), so the assumption is false by construction.
ad hoc to paper The learnable approximators h and g can estimate the relevant conditional distributions P(B) and P(V = v_c).
The architectures of h and g are not specified, and they are simply trained end-to-end to maximize classification accuracy, with no separate causal loss.

invented entities (3)

Cross-modal bias B
purpose: Latent confounder representing spurious visual-textual correlations in pretrained VLMs.
No falsifiable handle is provided; it is represented by an attention-weighted aggregation of visual features.
Visual confounder Z
purpose: Latent confounder representing fine-grained visual elements such as lighting and attire.
No measurement or falsifiable prediction; its removal is claimed via the front-door adjustment using M.
Visual mediator M (set equal to debiased text T')
purpose: Mediator in front-door adjustment to block the backdoor path from Z.
M is defined as the model's own debiased text embedding, providing no independent evidence of causal status.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cross-Modal Dual-Causal Learning for Long-Term Action Recognition." pith.science (2026). https://pith.science/paper/44XQNZFD

@misc{pith2026250706603,
  author       = {Pith},
  title        = {Pith review of: Cross-Modal Dual-Causal Learning for Long-Term Action Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/44XQNZFD}},
  note         = {Machine review of arXiv:2507.06603}
}

read the original abstract

Long-term action recognition (LTAR) is challenging due to extended temporal spans with complex atomic action correlations and visual confounders. Although vision-language models (VLMs) have shown promise, they often rely on statistical correlations instead of causal mechanisms. Moreover, existing causality-based methods address modal-specific biases but lack cross-modal causal modeling, limiting their utility in VLM-based LTAR. This paper proposes \textbf{C}ross-\textbf{M}odal \textbf{D}ual-\textbf{C}ausal \textbf{L}earning (CMDCL), which introduces a structural causal model to uncover causal relationships between videos and label texts. CMDCL addresses cross-modal biases in text embeddings via textual causal intervention and removes confounders inherent in the visual modality through visual causal intervention guided by the debiased text. These dual-causal interventions enable robust action representations to address LTAR challenges. Experimental results on three benchmarks including Charades, Breakfast and COIN, demonstrate the effectiveness of the proposed model. Our code is available at https://github.com/xushaowu/CMDCL.

Figures

Figures reproduced from arXiv: 2507.06603 by the authors.

**Figure 2.** The illustration of structural causal models (SCMs). [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The proposed causal graph of causal intervention in textual and visual modalities. The solid lines represent direct [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The framework of CMDCL. CMDCL leverages pretrained VLMs for cross-modal knowledge extraction and employs [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Visual analysis of the deconfounding effect. (a) and (b) present attention maps for the long-term actions “make [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Effection of hyperparameters. (a) Effection of the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 6.** Figure 6: Visualization of class-to-class co-classification prob [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 37 canonical work pages

[1]

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding?. InICML, Vol. 2. 4

work page 2021
[2]

Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. Inproceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308

2017
[3]

Long Chen, Yuhang Zheng, Yulei Niu, Hanwang Zhang, and Jun Xiao. 2023. Counterfactual samples synthesizing and training for robust visual question answering.IEEE Transactions on Pattern Analysis and Machine Intelligence45, 11 (2023), 13218–13234

work page 2023
[4]

Yifei Chen, Dapeng Chen, Ruijin Liu, Sai Zhou, Wenyuan Xue, and Wei Peng. 2024. Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18688–18698

work page 2024
[5]

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. 2021. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 6824– 6835

2021
[6]

Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell

work page
[7]

Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. 2024. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13504–13514

2024
[8]

Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. 2019. Time- ception for complex action recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 254–263

work page 2019

Show all 51 references

[9]

Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. 2019. Video- graph: Recognizing minutes-long human activities in videos.arXiv preprint arXiv:1905.05143(2019)

2019 arXiv
[10]

Md Mohaiminul Islam and Gedas Bertasius. 2022. Long movie clip classifica- tion with state-space video models. InEuropean Conference on Computer Vision. Springer, 87–104

2022
[11]

Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. 2023. Efficient movie scene detection using state- space transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18749–18758

2023
[12]

Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. 2019. Stm: Spatiotemporal and motion encoding for action recognition. InProceedings of the IEEE/CVF international conference on computer vision. 2000–2009

2019
[13]

Hilde Kuehne, Ali Arslan, and Thomas Serre. 2014. The language of actions: Recov- ering the syntax and semantics of goal-directed human activities. InProceedings of the IEEE conference on computer vision and pattern recognition. 780–787

2014
[14]

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

2023
[15]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning. PMLR, 12888–12900

2022
[16]

Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. 2024. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision. Springer, 237–255

2024
[17]

Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. 2022. Learning to recognize procedural activities with distant supervision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13853–13863

2022
[18]

An-An Liu, Chenxi Huang, Ning Xu, Hongshuo Tian, Jing Liu, and Yongdong Zhang. 2023. Counterfactual visual dialog: Robust commonsense knowledge learning from unbiased training.IEEE Transactions on Multimedia26 (2023), 1639–1651

2023
[19]

Hong Liu, Yongqing Sun, Yukihiro Bandoh, Masaki Kitahara, and Shin’ichi Satoh
[20]

Yang Liu, Guanbin Li, and Liang Lin. 2023. Cross-modal causal relational rea- soning for event-level visual question answering.IEEE Transactions on Pattern Analysis and Machine Intelligence45, 10 (2023), 11624–11641

2023
[21]

Yang Liu, Guanbin Li, and Liang Lin. 2023. Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering.IEEE Transactions on Pattern Analysis and Machine Intelligence(2023)

2023
[22]

Yang Liu, Fang Liu, Licheng Jiao, Qianyue Bao, Lingling Li, Yuwei Guo, and Puhua Chen. 2024. A knowledge-based hierarchical causal inference network for video action recognition.IEEE Transactions on Multimedia(2024)

2024
[23]

Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. 2022. Causality inspired representation learning for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8046–8056

2022
[24]

Anindya Mondal, Sauradip Nag, Joaquin M Prada, Xiatian Zhu, and Anjan Dutta
[25]

Leland Gerson Neuberg. 2003. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000.Econometric Theory19, 4 (2003), 675–685

2003
[26]

Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji- Rong Wen. 2021. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12700–12710

2021
[27]

arXiv preprint arXiv:2307.10763(2023)

MSQNet: Actor-agnostic Action Recognition with Multi-modal Query. arXiv preprint arXiv:2307.10763(2023)

2023 arXiv
[28]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learnin...

2021
[29]

Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. Tokenlearner: What can 8 learned tokens do for images and videos?arXiv preprint arXiv:2106.11297(2021)

2021 arXiv
[30]

2016.Causal inference in statistics: a primer

Judea Pearl. 2016.Causal inference in statistics: a primer. John Wiley & Sons

2016
[31]

Bernhard Schölkopf. 2022. Causality for machine learning. InProbabilistic and causal inference: The works of Judea Pearl. 765–804

2022
[32]

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021. Toward causal repre- sentation learning.Proc. IEEE109, 5 (2021), 612–634

2021
[33]

Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. 2019. As- semblenet: Searching for multi-stream neural connectivity in video architectures. arXiv preprint arXiv:1905.13209(2019)

2019 arXiv
[34]

Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. 2025. On the utility of 3d hand poses for action recognition. InEuropean Conference on Computer Vision. Springer, 436–454

2025
[35]

Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 201...

2016
[36]

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. InProceedings of the IEEE interna- tional conference on computer vision. 618–626

2017
[37]

2001.Causation, prediction, and search

Peter Spirtes, Clark Glymour, and Richard Scheines. 2001.Causation, prediction, and search. MIT press

2001
[38]

Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. Coin: A large-scale dataset for comprehensive in- structional video analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1207–1216

2019
[39]

Zeen Song, Siyu Zhao, Xingyu Zhang, Jiangmeng Li, Changwen Zheng, and Wen- wen Qiang. 2024. Learning Invariant Causal Mechanism from Vision-Language Models.arXiv preprint arXiv:2405.15289(2024)

2024 arXiv
[40]

Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. 2023. Selective structured state-spaces for long-form video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6387–6397

2023
[41]

Mengmeng Wang, Jiazheng Xing, and Yong Liu. 2021. Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472(2021)

2021 arXiv
[42]

Junyan Wang, Yiqi Jiang, Yang Long, Xiuyu Sun, Maurice Pagnucco, and Yang Song. 2023. Deconfounding causal inference for zero-shot action recognition. IEEE Transactions on Multimedia(2023)

2023
[43]

Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. 2023. Bidirectional cross-modal knowledge exploration for video recog- nition with pre-trained vision-language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recogn...

2023
[44]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. InProceedings of the 32nd International Conference on Machine Learning (Proc...

2015
[45]

Wenhao Wu, Zhun Sun, Yuxin Song, Jingdong Wang, and Wanli Ouyang. 2024. Transferring vision-language models for visual recognition: A classifier perspec- tive.International Journal of Computer Vision132, 2 (2024), 392–409

2024
[46]

Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction- tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858(2023). Conference’17, July 2017, Washington, DC, USA Xu et al

2023 arXiv
[47]

Jiaming Zhou, Kun-Yu Lin, Haoxin Li, and Wei-Shi Zheng. 2021. Graph-based high-order relation modeling for long-term action recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8984–8993

2021
[48]

Tianshu Yu, Yikang Li, and Baoxin Li. 2020. Rhyrnn: Rhythmic rnn for recognizing events in long and complex videos. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16. Springer, 127–144

2020
[51]

Jiaming Zhou, Kun-Yu Lin, Yu-Kun Qiu, and Wei-Shi Zheng. 2023. Twinformer: Fine-to-coarse temporal modeling for long-term action recognition.IEEE Trans- actions on Multimedia(2023)

2023
[2017]

InProceedings of the IEEE conference on computer vision and pattern recognition

Actionvlad: Learning spatio-temporal aggregation for action classification. InProceedings of the IEEE conference on computer vision and pattern recognition. 971–980
[2023]

Deep Counterfactual Representation Learning for Visual Recognition against Weather Corruptions.IEEE Transactions on Multimedia(2023)

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding?. InICML, Vol. 2. 4

work page 2021

[2] [2]

Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. Inproceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308

2017

[3] [3]

Long Chen, Yuhang Zheng, Yulei Niu, Hanwang Zhang, and Jun Xiao. 2023. Counterfactual samples synthesizing and training for robust visual question answering.IEEE Transactions on Pattern Analysis and Machine Intelligence45, 11 (2023), 13218–13234

work page 2023

[4] [4]

Yifei Chen, Dapeng Chen, Ruijin Liu, Sai Zhou, Wenyuan Xue, and Wei Peng. 2024. Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18688–18698

work page 2024

[5] [5]

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. 2021. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 6824– 6835

2021

[6] [6]

Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell

work page

[7] [7]

Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. 2024. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13504–13514

2024

[8] [8]

Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. 2019. Time- ception for complex action recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 254–263

work page 2019