{"id":"9049cbb9-0719-4c7f-ab70-3d7e3fff152a","arxiv_id":"2412.09895","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"STDD improves zero-shot action recognition by adding a parameter-free space-time cross attention to CLIP and augmenting text prompts with an LLM-built action semantic knowledge graph.","lead":"This paper presents STDD, a CLIP-based method for zero-shot action recognition that adds a space-time cross attention to the vision encoder and generates detailed spatial and temporal text prompts using an LLM knowledge graph. The authors report state-of-the-art accuracy on UCF101, HMDB51, and Kinetics-600, though the gains over prior work are mostly small.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no increased computational complexity' claim is internally inconsistent: Eq. (7) adds an extra MHSA pass per temporal scale, so STCA costs about 1.5x spatial-only attention at r=0.5, S=2.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the paper claims no additional computational cost, but the architecture appears to compute MHSA twice per block. This is the most concrete and internally checkable weakness in the central claim, which has two parts: state-of-the-art accuracy and no-extra-compute temporal modeling. The accuracy margins over previous methods are often small and sometimes within one standard deviation, and the training hyperparameters are not fully disclosed, but those are statistical and reproducibility concerns rather than a definite inconsistency. By contrast, the complexity claim can be settled directly from the equations: Eq. (1) and Eq. (7) both call MHSA, and with the paper's own hyperparameters the extra pass costs about 50% of the spatial attention. I do not see a reason to reject the accuracy claim on the evidence presented; the benchmarks and protocols are standard and the code is promised. The appropriate outcome remains a conditional acceptance, pending a corrected or qualified complexity statement. The FLOP-count test is simple and decisive, so the concern is not speculative.","tokens_in":20679,"tokens_out":8992,"duration_ms":101654,"concrete_test":"Run the released STDD code with ViT-B/16, r=0.5, S=2 and count attention FLOPs, or measure wall-clock time per forward pass, for one ViT block with and without the Space-time Cross Attention, holding all other settings fixed. If the ratio is approximately 1.5 (or measurably greater than 1), the 'equal to spatial-only' claim is refuted. Report the measured ratio and revise the complexity statement accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's efficiency claim rests on the asymptotic expression in Section 3, 'Computational Complexity': O(T N^2 + S T (N')^2) = O(T N^2), which is called 'equal to that of spatial-only attention.' However, Eq. (1) already applies MHSA to the full T x (N+1) token set, and Eq. (7) applies the same MHSA again to the N' masked/mixed tokens for each of the S temporal scales before average pooling. With the implemented N' = r N (r = 0.5) and S = 2 (scales [±1, ±2]), the attention FLOPs per block are (1 + S r^2) times spatial-only, i.e. approximately 1.5x, plus additional LayerNorm operations, residual additions, and memory for multiple mixed-token tensors. The Big-O notation hides this constant factor, so the statement 'without increasing computational complexity' is not accurate in an actual-compute sense. This does not invalidate the reported accuracy gains, but it undercuts a central selling point of the method and the comparison against AIM and full space-time attention. The complexity claim should be corrected or explicitly qualified as asymptotic with a non-negligible constant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STDD, a CLIP-based framework for zero-shot action recognition. On the vision side, it augments the spatial self-attention of each ViT block with a four-step operation (Window Shift Masking, Multi-scale Channel Mixing, Spatial Padding, and Short-cutting) to capture cross-frame dynamics without adding parameters. On the text side, it constructs an Action Semantic Knowledge Graph (ASKG) by prompting GPT-3.5 to generate spatial and temporal text prompts. During training, frame-level video representations are aligned with prompt-level text representations and regularized by feature distillation from frozen CLIP. Experiments on UCF101, HMDB51, and Kinetics-600 under three evaluation protocols are reported against prior CLIP-based ZSAR methods.","tokens_in":20976,"tokens_out":4411,"duration_ms":42362,"significance":"The paper's main strength is the systematic combination of a parameter-free temporal attention mechanism with structured LLM-based semantic augmentation, evaluated under a clean K400-training/novel-class protocol. The public code and the detailed appendices, including full LLM prompts and ablations of masking and temporal scales, are useful for reproducibility. If the efficiency claim were accurate, the method would be practically attractive. However, the efficiency claim hides a non-trivial constant factor, and several improvements over prior state-of-the-art are within one standard deviation; both issues are addressable.","major_comments":[{"comment":"The claim that STCA increases no computational complexity rests on the asymptotic expression O(T N^2 + S T (N')^2) = O(T N^2). Asymptotically this is true, but the paper's stronger claim 'without increasing computational complexity' is not supported: Eq. (7) performs a second MHSA on N' tokens for each of S temporal scales, in addition to the full MHSA of Eq. (1). With the implemented r=0.5 and S=2 (scales [±1, ±2]), the attention FLOPs are about (1 + S r^2) = 1.5x spatial-only, plus extra LayerNorm, residual additions, and memory. Please qualify the claim as asymptotic and report actual FLOPs or runtime, or revise the comparison against AIM and full space-time attention.","section":"Abstract / §3 Computational Complexity"},{"comment":"The final training objective is defined only as L_CE 'together with feature distillation loss proposed by (Huang et al. 2024)'; the loss is not defined, its weighting is not given, and no ablation isolates its contribution. Since this term is part of the optimization, the reported results are not fully reproducible. Please provide the loss expression, the hyperparameter value, and an ablation with and without this term.","section":"§3 Training Objectives"},{"comment":"The channel-index hyperparameter gamma in Eq. (5), where d_delta = gamma * D, is never reported. It controls how much channel information is mixed at each temporal scale and should be listed in Implementation Details, ideally with an ablation.","section":"§3 Multi-scale Channel Mixing / Implementation Details"},{"comment":"The statement that STDD 'consistently surpasses state-of-the-art' is weakened by the reported variances: the gains over the previous best on UCF (85.2 +/- 1.2 vs. 84.8 +/- 1.1), K600 (75.1 +/- 0.7 vs. 74.8 +/- 0.9), and HMDB (55.9 +/- 0.2 vs. 55.6 +/- 1.4) are all within one standard deviation. Please report multiple seeds or statistical significance, or soften the claim accordingly.","section":"Table 2 / Main Results"}],"minor_comments":[{"comment":"The text refers to 'our MSM' when discussing the masking strategy, but the method is called WSM; this should be corrected.","section":"Supplementary Material, Table 5"},{"comment":"The periodic function phi(·|·) that generates masking maps is described only in words; the appendix would benefit from pseudocode or an exact procedural definition.","section":"Eq. (3)"},{"comment":"The caption uses the term 'Window view' without defining the coordinate axes or the meaning of the numbers (1 and 3); please make the figure self-contained.","section":"Figure 3(c)"},{"comment":"In the Related Work section on semantic knowledge, one citation is rendered as '?' rather than an actual reference; this needs to be fixed.","section":"References"},{"comment":"The abbreviation 'HMBD' appears in the caption while the text uses 'HMDB51'; please use a single consistent abbreviation.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The main technical issue is the overstated efficiency claim, which is a presentation and reproducibility matter rather than a fatal flaw. The accuracy comparison would benefit from significance testing, but the paper's core ideas are sound and within scope for a computer vision venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take on Yu et al. (arXiv:2412.09895). It is a decent empirical paper that does what it says on the tin. The new bits are the STCA block—masking, channel mixing, padding, shortcut applied around the existing CLIP spatial attention—and the ASKG text augmentation, which builds a structured graph of static vs dynamic prompts. Neither is a conceptual breakthrough; both are recombinations of known ideas (MAR masking, TSM/X-ViT shifts, LLM-prompt augmentation). But the ablations are thorough: they show each component helps, they compare masking strategies and temporal scales, and they demonstrate that the text augmentation transfers to other backbones (Text4Vis with their prompts improves). That is solid, reproducible work ethic. The code is linked, and the experimental setup is clean: K400 training, disjoint UCF/HMDB/K600 classes, LLM prompts generated from class names. No leaks. Credit where due.\n\nThe soft spots are real but not fatal. First, the efficiency claim is over-sold. The complexity section writes O(T N^2 + S T (N')^2) = O(T N^2), which is true asymptotically, but with r=0.5 and S=2 the actual attention FLOPs per block are about 1.5x spatial-only. Saying 'without increasing computational complexity' is a fair description in the Big-O sense, but not in an actual-compute sense, and the paper uses the phrase to contrast with AIM and full space-time attention. I'd push to rephrase or at least report measured FLOPs/runtime. Second, the training details are thin: the distillation loss is cited to (Huang et al. 2024) but the exact form and weight aren't given; gamma in d_delta = gamma*D is not disclosed. This is easily fixed but currently hampers reproduction. Third, the gains over the previous best are small—0.4% on UCF EP3, 0.3% on K600—and the reported standard deviations overlap. 'Consistently surpasses' is true on the means, but the statement is stronger than the evidence supports. Also, there is an unresolved '?' in the related work citation list.\n\nWould I send this to a serious referee? Yes. The accuracy claim is plausible, the experiments are carefully done, and the limitations are acknowledged (LLM hallucination, absence of visual cues in text augmentation). A referee should ask for the complexity claim to be corrected, the missing hyperparameters and distillation formulation to be provided, and ideally for a runtime comparison. But the core contribution—a parameter-efficient way to add temporal modeling to CLIP with structured text augmentation—is real, and the reported numbers are new measurements. This is a legitimate paper for the community, not a desk reject.","headline":"Competent, well-ablated ZSAR paper with a real but fixable efficiency-claim problem; the accuracy gains are small, yet the work is better than the usual SOTA churn and deserves a serious referee.","tokens_in":21506,"tokens_out":3035,"would_cite":true,"duration_ms":31823,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding masked channel mixing around CLIP's attention, plus LLM-built spatial and temporal prompts, yields state-of-the-art zero-shot action recognition on UCF101, HMDB51, and Kinetics-600.","keywords":["zero-shot action recognition","CLIP","Space-time Cross Attention","Action Semantic Knowledge Graph","spatiotemporal text augmentation","video understanding","large language models","contrastive learning"],"falsifier":"Time and profile STDD against its own spatial-only baseline on the same 8-frame clips: if Eq. (7) executes a second multi-head self-attention on the half-masked tokens, total attention FLOPs should be about $(1 + S r^2)$ times spatial-only (with $S=2$, $r=0.5$, roughly 1.5x), and the measured wall-clock or FLOP ratio will show whether the 'no increased computational complexity' claim holds. Separately, zero-shot accuracy on a held-out split with and without Space-time Cross Attention would show how much of the reported gain comes from the temporal mechanism rather than from text augmentation.","tokens_in":20504,"feed_emoji":"🎬","tokens_out":9121,"duration_ms":82614,"temperature":0.7,"pith_summary":"Zero-shot action recognition asks a model to label video actions it never saw during training, so the model must rely on general visual and semantic knowledge. This paper claims that CLIP, a strong image-text model, can be turned into a spatiotemporal action expert without adding any trainable parameters on the vision side: within each transformer block, a masked, channel-swapped set of tokens is passed through the existing spatial attention, then padded and shortcut back, letting the network see cross-frame motion cues. On the text side, the paper builds an Action Semantic Knowledge Graph by prompting a large language model for each action's objects, sub-actions, and their order, then turns those relations into separate spatial and temporal text prompts. Training aligns frame-level video features with these prompt-level text features, while a frozen CLIP distills its general visual knowledge into the video encoder. On Kinetics-600, UCF101, and HMDB51 under zero-shot protocols, the paper reports consistent gains over prior CLIP-based methods.","feed_headline":"Mask, mix, pad lifts CLIP past zero-shot video rivals","feed_subtitle":"A frame-level attention rewrite plus LLM-built action prompts raises top-1 accuracy on UCF101, HMDB51, and Kinetics-600.","key_machinery":"The load-bearing object is Space-time Cross Attention (STCA), a four-step rewrite of CLIP's existing spatial attention. Before the multi-head self-attention, (1) Window Shift Masking drops a fraction of spatial tokens with a mask that shifts along time; (2) Multi-scale Channel Mixing swaps channel slices between the surviving tokens and their neighbors at several temporal offsets, reusing the established 'shift trick' for zero-cost motion exchange. After attention, (3) Spatial Padding restores the original token positions and (4) a residual Short-cutting adds the mixed temporal information back into the main stream. The companion machinery is the Action Semantic Knowledge Graph (ASKG), an LLM-generated graph in which action categories connect to object nodes and sub-action nodes via relations such as 'used to shoot' and 'precedes'; parsing this graph yields the spatial prompts [S] and temporal prompts [T] that the contrastive alignment uses as fine-grained text targets.","core_discovery":"The central claim is that the two main weaknesses of directly finetuning CLIP for zero-shot action recognition—missing temporal dynamics and overly coarse class-level text—can be fixed simultaneously. The vision fix, Space-time Cross Attention, applies window-shift masking and multi-scale channel mixing to a fraction of the spatial tokens before the existing multi-head self-attention, then pads and residual-shortcuts the result, so temporal information flows through the same parameters CLIP already has. The text fix, spatiotemporal text augmentation from an Action Semantic Knowledge Graph, generates prompts that name the objects involved in an action (spatial prompts) and the ordered sub-actions that compose it (temporal prompts), giving the contrastive alignment finer-grained anchors than a single class name. With these two pieces, the paper claims state-of-the-art zero-shot accuracy on UCF101, HMDB51, and Kinetics-600, and shows the text prompts alone can also lift a baseline model's accuracy when swapped in without retraining.","pith_inferences":["A natural stress test is to measure wall-clock latency and FLOPs on identical hardware: since the mixed-token MHSA of Eq. (7) is a second attention pass, the exact cost is roughly $(1 + S r^2)$ times the spatial-only attention at the same token count; the 'no increased computational complexity' claim is asymptotic and may not match practical timing.","The paper's own limitation note—LLM hallucination makes prompt quality vary per category—suggests a testable extension: ground the generated prompts in the video's actual detected objects, or filter low-confidence LLM relations, to reduce variance across action classes.","The method aligns frame-level features to prompt-level text; that design should transfer to few-shot action recognition, where a handful of labeled examples could replace or calibrate the LLM prompts, and to open-vocabulary action spotting, where the same score could localize when an unseen action occurs.","Because the text prompts decompose actions into ordered sub-actions, the learned alignment might support action anticipation—predicting the next sub-action from the video frames—without any new supervision, a consequence the paper does not draw."],"forward_implications":["If the accuracy gains hold, STDD becomes the new state of the art for CLIP-based zero-shot action recognition on UCF101, HMDB51, and Kinetics-600 under the standard protocols.","Because the vision-side change adds no parameters, the same Space-time Cross Attention can be dropped into other ViT-based video models, potentially improving action localization or video captioning without retraining a custom architecture.","The ASKG prompt set is generated offline by an LLM, so it can be extended to new action categories simply by prompting, without collecting new video data or updating the text encoder.","The frame-to-prompt fine-grained alignment used for training also serves as the zero-shot inference score, so the method needs no extra classifier or video-level aggregation at test time beyond averaging frame-prompt matches."],"supporting_citations":[{"why":"Supplies the pretrained CLIP vision and text encoders whose spatial attention STDD rewrites and whose alignment is the base for zero-shot scoring.","marker":"Radford et al. 2021"},{"why":"Supplies the space-time mixing 'shift trick' that Multi-scale Channel Mixing extends to masked, interleaved spatial tokens.","marker":"Bulat et al. 2021"},{"why":"Supplies the masking strategy that Window Shift Masking adapts from masked reconstruction to inter-frame token selection.","marker":"Qing et al. 2023"},{"why":"Supplies the frozen-CLIP feature distillation loss that regulates the learned video representations for generalization.","marker":"Huang et al. 2024"},{"why":"Supplies the temporal attention-view baseline and the epoch-averaging protocol used for evaluation.","marker":"Weng et al. 2023"},{"why":"The spatiotemporal-descriptor text-augmentation baseline whose ASKG prompts outperform on UCF101 and Kinetics-600.","marker":"Chen et al. 2024"},{"why":"Supplies the Kinetics-600 zero-shot evaluation splits and the X-CLIP baseline.","marker":"Ni et al. 2022"},{"why":"The Text4Vis baseline whose category prompts are replaced by ST Aug. in an adaptivity experiment, and whose alignment objective is followed.","marker":"Wu, Sun, and Ouyang 2023"},{"why":"Supplies the EP 1 and EP 2 zero-shot evaluation protocols used for UCF101 and HMDB51.","marker":"Brattoli et al. 2020"},{"why":"The GPT-3.5 language model that the ASKG construction and text-prompt generation prompt for static and dynamic action knowledge.","marker":"Achiam et al. 2023"}],"fun_headline_variants":["STDD: mask, mix, pad + action graph lift CLIP for zero-shot video","Space-time attention and action prompts push CLIP past zero-shot rivals","STDD: spatiotemporal dynamic duo makes CLIP zero-shot action-ready","Rewiring CLIP's attention and text beats zero-shot action baselines","CLIP gains motion sense and fine-grained text via STDD for ZSAR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the extra attention pass over the masked, channel-mixed tokens costs no meaningful compute, so the total stays 'equal to spatial-only attention'; if that extra pass is a real second forward step, the no-complexity increase selling point weakens.","fun_headline_variants_meta":{"raw":{"variants":["STDD: mask, mix, pad + action graph lift CLIP for zero-shot video","Space-time attention and action prompts push CLIP past zero-shot rivals","STDD: spatiotemporal dynamic duo makes CLIP zero-shot action-ready","Rewiring CLIP's attention and text beats zero-shot action baselines","CLIP gains motion sense and fine-grained text via STDD for ZSAR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1731,"prompt_tokens":1003,"completion_tokens":728,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":628}},"tokens_in":619,"tokens_out":728,"duration_ms":7955,"temperature":1.0,"reasoning_tokens":628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:36:39.599006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Time and profile STDD against its own spatial-only baseline on the same 8-frame clips: if Eq. (7) executes a second multi-head self-attention on the half-masked tokens, total attention FLOPs should be about $(1 + S r^2)$ times spatial-only (with $S=2$, $r=0.5$, roughly 1.5x), and the measured wall-clock or FLOP ratio will show whether the 'no increased computational complexity' claim holds. Separately, zero-shot accuracy on a held-out split with and without Space-time Cross Attention would show how much of the reported gain comes from the temporal mechanism rather than from text augmentation.","supporting_citations":[],"review_version":1}