{"id":"bfa5afa6-cdcb-4d27-adf9-e3447759a273","arxiv_id":"1908.00497","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A plug-in cross-modality attention block for two-stream video classification is shown to improve accuracy by about one percentage point, though the experimental design does not isolate the block's contribution from a different training procedure.","lead":"This paper proposes a cross-modality attention block that lets one video stream (RGB) attend over the feature maps of another stream (optical flow) at intermediate layers. The authors report accuracy gains over classic two-stream fusion on Kinetics and UCF-101, but the gains are modest and the experiments are confounded.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed CMA gain is unidentifiable: CMA iter1-R is trained with a Kinetics-pretrained, frozen flow branch, but no no-CMA baseline uses the same protocol; the +0.96% over two-stream may be a training-scheme artifact.","rationale":"The reader's core concern is correct: the paper lacks a controlled ablation that isolates the CMA block from the iterative pretrained-frozen-flow training protocol. However, the reader's characterization of the two-stream baseline as 'trained jointly' is inaccurate; Section 5.3 describes the iter0 baseline as having two branches trained independently, consistent with the original two-stream approach. This factual point does not weaken the central objection. The reported accuracy gaps are small, no significance testing is provided, and every comparison (Tables 2, 4, 5) changes the training scheme together with the attention module. The visualizations are qualitative and do not establish a causal role for the attention operation. Therefore the current submission does not support the abstract's claim of 'strong performance superiority' for the CMA block itself. A controlled no-CMA baseline under the same iterative protocol is the minimal experiment needed; if it closes the gap, the paper's central claim fails, and if it does not, the CMA block would be supported. The reader's REJECT verdict remains appropriate for the submitted version.","tokens_in":12368,"tokens_out":6038,"duration_ms":64656,"concrete_test":"Train a control 'no-CMA two-stream iter1' with the exact Section 4.3 protocol: pretrain the flow ResNet-50 on Kinetics, freeze it, then train the RGB ResNet-50 (same ImageNet initialization, epochs, learning-rate schedule, augmentation, TSN segments) with all CMA blocks removed or replaced by identity. Evaluate top-1 on the Kinetics validation set with three seeds. If this control reaches or exceeds 72.17%, the CMA block adds nothing beyond the pretrained-frozen-flow training scheme; if it stays near the 67.73% RGB-only baseline, the attention mechanism is responsible for the gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 defines an iterative training scheme: the flow branch is first trained on Kinetics, then frozen while the RGB branch is trained at iter1. Table 2 attributes CMA iter1-R's 72.17% top-1 to the cross-modality attention block, but no baseline trains an RGB branch under the same pretrained-frozen-flow protocol without CMA blocks. The iter0 two-stream baseline is trained independently (as stated in Section 5.3), so the comparison conflates two changes: the attention mechanism and the pretraining/freezing/iterative-training procedure. The same gap appears in Table 4: the non-local RGB+Flow model is not trained with the iterative frozen-flow scheme, so its 71.67% is not a controlled comparator. Because the reported differences are small (roughly 0.5–1%) and no error bars or multiple-seed results are given, the observed gains cannot be causally attributed to the CMA block. The 3D-CMA experiments in Table 5 show the same pattern: the Flow column is identical to P3D, and only the RGB branch is trained with CMA, again without a no-CMA control. The residual identity initialization is a reasonable design choice, but it does not replace an ablation that isolates the attention operation from the training protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cross-modality attention (CMA) block that lets one modality branch (e.g., RGB) attend over positions in another modality branch (e.g., optical flow), in contrast to self-attention non-local blocks. The block is designed as a residual wrapper and can be inserted into existing two-stream or 3D CNN architectures. The authors report experiments on Kinetics and UCF-101, claiming that CMA blocks improve accuracy over two-stream baselines and non-local networks. The central quantitative claim is that CMA iter1-R reaches 72.17% top-1 on Kinetics compared with the two-stream baseline's 71.21%, while using fewer parameters (Table 2). The paper also includes attention visualizations and a transfer-learning experiment to UCF-101.","tokens_in":12742,"tokens_out":3902,"duration_ms":40431,"significance":"If the empirical claim were cleanly established, the cross-modality attention idea would be a useful and well-motivated contribution to video classification: it generalizes non-local self-attention to cross-modal attention, is architecture-agnostic, and the visualizations suggest interpretable behavior. The paper also demonstrates compatibility with both 2D and 3D backbones, and the residual design with zero-initialized BN is a sensible implementation choice. However, the evidence as presented does not isolate the effect of the CMA block from the accompanying change in training strategy, so the significance of the claimed gains is currently not established.","major_comments":[{"comment":"The main claim that the CMA block improves accuracy is confounded by a difference in training protocol. The baseline two-stream (iter0) is trained with both branches trained independently (per the footnote in §5.3), whereas CMA iter1-R is obtained after training the RGB branch while the flow branch is frozen and pretrained on Kinetics. No no-CMA baseline is run under the same iterative pretrained-frozen-flow protocol. Therefore the reported +0.96% (72.17 vs. 71.21) cannot be causally attributed to the cross-modality attention block; it may be an artifact of the pretraining/freezing scheme. A controlled ablation (e.g., two-stream with the same frozen flow branch and iterative training but without CMA blocks) is required to support the paper's central claim.","section":"§4.3, §5.3, Table 2"},{"comment":"The comparison with non-local networks is also uncontrolled. The non-local RGB+Flow model is trained without the iterative frozen-flow scheme, while the CMA models use that scheme, so the accuracy gap (71.67 vs. 72.17 or 72.62) cannot be assigned to the attention mechanism. In addition, the non-local blocks are added only to the RGB branch, whereas the CMA model includes blocks in both branches (even if the flow branch is frozen during iter1), making the architectural comparison asymmetric. The conclusion that CMA outperforms non-local is therefore not justified by the presented experiments.","section":"§5.5, Table 4"},{"comment":"The 3D-CMA experiments suffer from the same missing control. In Table 5, the Flow column is identical to the P3D baseline, and only the RGB branch is augmented with a single CMA block and then partially fine-tuned. There is no no-CMA P3D model fine-tuned with the same layer-wise protocol (training only the last block and following layers), so the improvement in the RGB column (e.g., 74.41 vs. 70.98 for 12 frames) could stem from the fine-tuning scheme rather than the CMA block. A matched baseline is needed to attribute the gain.","section":"§5.6, Table 5"},{"comment":"All reported accuracy differences are small (roughly 0.5–1%) and no error bars or multiple-seed results are provided. Given the known variance in training deep video models on Kinetics and the sensitivity to hyperparameters (including the fusion weights selected on the validation set in §5.2), the reported gaps are not shown to be statistically meaningful even apart from the confounds above. The paper should report mean and standard deviation over multiple runs or otherwise justify that the differences exceed training noise.","section":"§5.2–§5.4"}],"minor_comments":[{"comment":"The text contains a typo: \"litter has been done\" should be \"little has been done.\"","section":"§2"},{"comment":"The footnote explaining that iter0 does not initialize the CMA model is helpful but confusingly worded; it should explicitly state that the baseline is trained with the standard two-stream protocol (both branches trained from scratch or with standard initialization) and not with the iterative scheme described in §4.3.","section":"§5.3 footnote"},{"comment":"The x-axis of Figure 3 is labeled \"Fusion Weight Ratio (RGB:Flow)\" but the actual tick values are not readable in the text version; please ensure the figure is legible and that the selected weights (5:1 and 1:5) are clearly marked.","section":"§5.2, Figure 3"},{"comment":"The sentence \"Figure 4 showed the top-20 most improved categories\" mixes tenses; also, the figure itself is not included in the text, so the reader cannot verify the claimed comparison. Please include the figure or refer to the appendix.","section":"§5.4"},{"comment":"The meaning of the \"modality\" column for CMA rows is ambiguous: the rows labeled \"RGB + Flow\" refer to the two-stream fusion output, but it would be clearer to indicate which branch output is being evaluated (e.g., \"CMA iter1-R\" is the RGB branch output).","section":"Table 4"},{"comment":"There is a typo: \"spatial aligned\" should be \"spatially aligned.\"","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the paper is clearly written, but the experimental evaluation as it stands does not support the central claim because of the confounded training protocol. The necessary fixes are additional controlled experiments: (i) a two-stream baseline trained with the same iterative frozen-flow protocol without CMA blocks, (ii) a fair non-local baseline with matched training and matched block placement, and (iii) multi-seed results with error bars. If the authors can provide these, the paper could become publishable; without them, the claimed superiority of CMA over two-stream is not established. I also note that the paper appears to be an early arXiv version; the authors should compare with more recent cross-modal attention work in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central claim is not shown. The paper proposes a cross-modality attention block that is exactly the standard non-local/self-attention operation with the query from one stream and keys/values from the other. That is a direct extension, and the authors are honest about it: when the two modalities are identical, CMA reduces to non-local. The residual wrapper and zero-initialized BN are clean, sensible choices, and the block is a plausible plug-in for two-stream video models. The experiments on Kinetics and UCF-101 with ResNet-50 and P3D are real work, and the qualitative attention maps are informative.\n\nThe soft spot is the experimental design, and the stress-test note is right. Section 4.3 defines an iterative training scheme: the flow branch is pretrained on Kinetics, frozen, then the RGB branch is trained at iter1. Table 2 then compares CMA iter1-R (72.17 top-1) against the two-stream baseline (71.21), but that baseline trains both branches independently. So the +0.96% conflates the CMA block with the pretraining/freezing protocol. The non-local comparison in Table 4 has the same issue: the RGB+Flow non-local model does not use the iterative frozen-flow scheme, so its 71.67 is not a controlled comparator. Since the margins are around 0.5-1% and there are no error bars or multiple seeds, the reported gain cannot be causally attributed to CMA. Table 5 repeats the pattern: the Flow column is identical to P3D, and the RGB branch gets CMA without a no-CMA control.\n\nThe fusion weight ratio (5:1 or 1:5) is selected from validation, which is standard hyperparameter tuning rather than circular reasoning. The UCF-101 transfer results are only fine-tuned on the last fc layer, so they are not directly comparable to numbers that fine-tune more, but the paper mostly uses them as a sanity check.\n\nWho is this for? Researchers working on multi-modal fusion and video action recognition. If you are looking for a plug-in attention module, this is a reasonable starting point, but I would not cite the accuracy gain as evidence until the confound is fixed. A revision with a no-CMA baseline trained under the same iterative protocol, and a couple of seeds, could make this a decent modest contribution. I would send it to peer review rather than desk-reject, but the referee should ask for those ablations.","headline":"The CMA gain is plausibly real but not demonstrated: the +0.96% is confounded with a different training protocol.","tokens_in":13138,"tokens_out":2634,"would_cite":false,"duration_ms":25700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-modality attention, which lets one video branch query the other's feature maps, beats late score fusion while using fewer parameters.","keywords":["video classification","cross-modality attention","two-stream networks","optical flow","action recognition","attention mechanism","non-local networks","temporal segment networks"],"falsifier":"Train the two-stream ResNet-50 baseline under the exact CMA schedule—flow branch pretrained on Kinetics and frozen, RGB branch trained in isolation—but with no CMA blocks; if it also reaches roughly 72% top-1 on Kinetics, the attention block is not what drives the gain. A second check is to verify that zeroing $W_{\\text{out}}$ in every CMA block recovers the baseline accuracy exactly, as the identity-initialization claim predicts.","tokens_in":12120,"feed_emoji":"🎬","tokens_out":7045,"duration_ms":71911,"temperature":0.7,"pith_summary":"This paper argues that the usual way of combining RGB and optical-flow streams in video classification—averaging their prediction scores at the very end—leaves useful cross-modal information on the table. It proposes a cross-modality attention (CMA) block that lets features in one branch query the other branch's feature maps at intermediate layers, so each modality can selectively pull in complementary information from the other. On Kinetics, the RGB branch of the CMA model reaches 72.17% top-1 accuracy with 1.8 times a ResNet-50's parameters, beating the 71.21% of a two-stream ResNet-50 that uses 2 times the parameters; using both branches gives 72.62%. The block is designed as a drop-in component for existing two-stream and 3D architectures, which is why the result would matter beyond this specific setting.","feed_headline":"Cross-modality attention beats late fusion in video classification","feed_subtitle":"A new attention block lets an RGB stream query the flow stream, gaining accuracy with fewer parameters.","key_machinery":"The central object is the CMA block, a wrapper around a cross-modality attention operation $z_i = \\frac{1}{C(x,y)} \\sum_j f(x_i,y_j) v(y_j)$ with $f(x_i,y_j) = \\exp(q(x_i) k(y_j)^T / \\sqrt{d_k})$, where $x$ and $y$ are feature maps from different branches. Queries come from one modality while keys and values come from the other, so each spatial position can absorb information from every position of the other modality. A $1\\times 1$ convolution $W_{\\text{out}}$ and a residual connection $+x_i$ wrap the operation, and the final batch-normalization scale is initialized to zero, making the block an identity mapping at initialization. That identity initialization is what lets it be inserted into any network without disrupting pretrained weights.","core_discovery":"The paper's central claim is that cross-modality attention, computed globally as a softmax-weighted sum over positions in the other modality, is a more effective fusion mechanism than late score averaging. The CMA operation follows the query-key-value form: a position in the RGB stream queries all positions in the flow stream, and the weighted flow features are added back into the RGB stream through a residual connection. The same operation applied in reverse lets the flow stream attend to RGB. Inserting five such blocks into a ResNet-50 at stages res3 and res4 yields a single RGB stream that outperforms the full two-stream baseline (72.17% versus 71.21% top-1 on Kinetics) at lower parameter cost, and the fusion of both streams reaches 72.62%. The paper also shows the block works when both inputs come from the same modality, in which case it becomes the non-local operation, and when inserted into a 3D P3D network.","pith_inferences":["Because the CMA model is trained with a different schedule—flow branch pretrained on Kinetics and frozen, branches updated iteratively—than the jointly trained two-stream baseline, the reported gain cannot be cleanly attributed to attention; a same-schedule baseline without CMA blocks would isolate the effect.","The attention maps' focus on moving, task-relevant objects suggests the learned attention could double as a motion-saliency prior for weakly supervised action localization, a use the paper does not develop.","The query-key-value formulation is modality-agnostic, so a natural extension is fusing RGB with audio, depth, or pose streams wherever their features can be spatially or temporally aligned.","The paper's reported failure mode—queries on background attending to background—points to a concrete refinement: a confidence or saliency gate on query positions could reduce wasted attention and is testable on Kinetics."],"forward_implications":["Hierarchical cross-modal attention can replace late score averaging without increasing parameter count: the RGB stream alone exceeds the two-stream fusion accuracy.","Because CMA reduces to the non-local operation when both inputs come from the same modality, one block type can supply both cross-modal and self-attention, making separate non-local blocks unnecessary.","CMA blocks transfer to 3D convolutional backbones: adding one block to P3D improves its accuracy whether the network is fed 12 or 16 frames.","On UCF-101 after Kinetics pretraining, fine-tuning only the final layer of the CMA model reaches 96.5% with a 2D backbone, approaching models that use 3D convolutions."],"supporting_citations":[{"why":"Defines the two-stream baseline and the late score-averaging fusion method that CMA is designed to improve.","marker":"[28]"},{"why":"Supplies the global non-local attention formulation that CMA extends, and the comparison model that CMA is measured against.","marker":"[38]"},{"why":"Provides the query-key-value attention notation and scaled dot-product form used in the CMA operation.","marker":"[34]"},{"why":"Supplies the ResNet-50 backbone, residual connections, and initialization scheme used in both branches.","marker":"[9]"},{"why":"Provides the Temporal Segment Network framework used for segment-based training and evaluation.","marker":"[37]"},{"why":"Provides the Kinetics dataset for the main experiments and for pretraining the flow branch.","marker":"[14]"},{"why":"Provides the TV-L1 optical flow algorithm used to compute the flow branch's input.","marker":"[41]"},{"why":"Provides the UCF-101 dataset used for transfer learning evaluation.","marker":"[30]"},{"why":"Provides the I3D model and two-stream variant that serve as comparison points, especially in the transfer-learning table.","marker":"[3]"}],"fun_headline_variants":["Cross-modality attention beats late fusion in video","Attention fusion outperforms two-stream late averaging","Query flow from RGB: attention wins in video classification","Single stream with cross-attention beats two-stream fusion","CMA block: better video fusion with fewer parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy gain comes from the CMA attention operation itself, yet the CMA model is trained with a different procedure than the two-stream baseline—pretraining and freezing the flow branch and training branches iteratively—so the training schedule alone could explain the improvement.","fun_headline_variants_meta":{"raw":{"variants":["Cross-modality attention beats late fusion in video","Attention fusion outperforms two-stream late averaging","Query flow from RGB: attention wins in video classification","Single stream with cross-attention beats two-stream fusion","CMA block: better video fusion with fewer parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1270,"prompt_tokens":907,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":523,"tokens_out":363,"duration_ms":3977,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:50:59.117355+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the two-stream ResNet-50 baseline under the exact CMA schedule—flow branch pretrained on Kinetics and frozen, RGB branch trained in isolation—but with no CMA blocks; if it also reaches roughly 72% top-1 on Kinetics, the attention block is not what drives the gain. A second check is to verify that zeroing $W_{\\text{out}}$ in every CMA block recovers the baseline accuracy exactly, as the identity-initialization claim predicts.","supporting_citations":[{"cited_title":"Simonyan and A","cited_arxiv_id":null,"evidence_quote":"Defines the two-stream baseline and the late score-averaging fusion method that CMA is designed to improve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the global non-local attention formulation that CMA extends, and the comparison model that CMA is measured against."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Provides the query-key-value attention notation and scaled dot-product form used in the CMA operation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Temporal Segment Network framework used for segment-based training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TV-L1 optical flow algorithm used to compute the flow branch's input."},{"cited_title":"Carreira and A","cited_arxiv_id":null,"evidence_quote":"Provides the I3D model and two-stream variant that serve as comparison points, especially in the transfer-learning table."}],"review_version":1}