{"id":"536efa70-dc1e-455d-98e1-286e6b5c5532","arxiv_id":"2412.20964","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HBI V2 models video-text alignment as a cooperative game with Hierarchical Banzhaf Interaction plus single/cross-modal representation fusion, improving retrieval, QA, and captioning benchmarks.","lead":"Video-language models usually only match whole videos to whole sentences. This paper adds a finer-grained signal by treating frames and words as cooperative game players and using Banzhaf Interaction to reward the pairs that cooperate, which improves retrieval, question answering, and captioning benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The characteristic-function conditions in §3.2.1 are asserted, not verified; the actual S in Eq. 12 may violate them, so Banzhaf targets may not encode the claimed semantic correspondence.","rationale":"The central claim is that supervising with Banzhaf Interaction computed from cross-modal similarity yields fine-grained semantic alignment. For this claim to be true, the Banzhaf values must actually reward strongly matched frame-word pairs and penalize mismatched ones. The paper assumes this through conditions (a)-(c), but does not verify them. The characteristic function S is only given by Eq. 12 for the full video-text pair; the reduction to coalitions and merged players is left unspecified. This is not just a missing proof, it is a definitional gap: without a rule for phi on subsets, Eq. 1 cannot be evaluated in the proposed framework, and the pretrained approximator's inputs ('the similarity matrix of video frames and text tokens') do not reveal how subsets are handled. The concern is internal (the paper's own definitions do not imply the conditions), not a disagreement with consensus. The ablation in Table 4 shows the Banzhaf loss contributes up to 0.8 R@1 / 1.3 QA, so if the target is meaningless the explanation for these gains is in doubt; however, the empirical results are plausible and code is promised, so the prior CONDITIONAL verdict remains appropriate. The reader identified the same weakest assumption, and the proposed enumeration test would settle whether the concern lands. Verdict unchanged.","tokens_in":26010,"tokens_out":6793,"duration_ms":66183,"concrete_test":"On a small subset of MSRVTT (e.g., 4 frames x 4 words), define phi(C) explicitly as Eq. 12 restricted to the tokens in coalition C, with the merged player [{i,j}] represented by the average of the frame and word embeddings. Enumerate all coalitions, compute exact Banzhaf Interaction from Eq. 1, and test condition (a) on caption-matched pairs and (b) on randomly mismatched pairs. Report the violation rate and whether I separates matched from mismatched pairs; if violations are frequent or separation is weak, the Banzhaf target is not a faithful alignment label. Also report the pretrained tiny model's error against these exact values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's fine-grained alignment signal depends on S in Eq. 12 satisfying conditions (a)-(c) of §3.2.1, so that Banzhaf Interaction from Eq. 1 measures correspondence of frame-word pairs. However, S is only defined for the full video/text; it is never defined for arbitrary coalitions or for the merged coalition [{i,j}] that Eq. 1 requires, and no proof or experiment shows conditions (a)-(c) hold. For S, a weighted average of per-frame/word max similarities, merging a strongly matched pair need not raise the payoff (the max can be attained elsewhere), and a mismatched pair need not lower it. If the inequalities fail, the I values in Eqs. 6-8 are not semantically meaningful, so the observed gains (Table 4) could arise from the extra prediction head, deep supervision, or self-distillation rather than from fine-grained Banzhaf alignment. The tiny-model approximation of I (Section 4.1) is also unvalidated, which compounds the risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes HBI V2, an extension of the authors' previous HBI method, for video-language representation learning. The method models video frames and text words as players in a cooperative game, defines a Banzhaf Interaction index over the cross-modal similarity function, and uses it as an auxiliary training target via a prediction head and KL divergence. It also introduces a representation reconstruction module that combines single-modal and cross-modal features, a hierarchical token merging scheme (entity/action/event levels), deep supervision, and self-distillation, together with task-specific heads for text-video retrieval, video question answering, and video captioning. Experiments on MSRVTT, ActivityNet Captions, DiDeMo, MSRVTT-QA, MSVD-QA, ActivityNet-QA, and MSRVTT captioning report consistent gains over prior methods, and ablations show each component contributes.","tokens_in":26307,"tokens_out":10319,"duration_ms":98139,"significance":"If the claimed effects are genuine, the paper would offer a broadly applicable recipe for adding fine-grained interaction modeling to contrastive video-language learning, with gains across retrieval, QA, and captioning. The code release, the multi-task evaluation, the ablations, and the efficiency measurements are concrete strengths. The main caveat is that the game-theoretic target is not yet shown to have the semantic content attributed to it: the defining conditions on the characteristic function are asserted rather than verified, and the computational approximation used in training is not validated. These are fixable in revision, but without such validation the 'Banzhaf Interaction' serving as fine-grained alignment signal could be just an additional self-supervision head.","major_comments":[{"comment":"The characteristic function φ is taken to be the similarity S of Eq. (12), but S is defined only for a complete video–text pair; it is never specified for an arbitrary coalition C⊆N or for the merged coalition N\\{i,j}∪{[{i,j}]}. Consequently the Banzhaf Interaction I([{v_i,t_j}]) in Eq. (1) is not well-defined with this choice of φ. Likewise, conditions (a)–(c) in Section 3.2.1 are asserted without proof or empirical check; for the max-over-frames form of Eq. (12), merging a strongly matched pair need not increase S (the maximum can be attained elsewhere), so condition (a) can fail. Because Eq. (8) optimizes against these I values, the central claim that the auxiliary loss provides fine-grained semantic alignment is not supported. The authors should either specify a coalition-level φ satisfying (a)–(c), or provide an empirical validation that the I values correlate with human-annotated or otherwise ground-truth frame–word correspondences.","section":"Section 3.2.1, Eq. (1) and Eq. (12)"},{"comment":"The pretrained tiny model that approximates Banzhaf Interaction is described in two sentences, with no information about how its training targets are generated (exact computation for small N? sampling?), how many instances, or what approximation error it achieves. Since the Banzhaf loss in Eq. (8) is computed from this approximation, the observed gains in Table 4 could in principle come from the extra prediction head rather than from the intended game-theoretic signal. Please report the approximation accuracy and, if feasible, an ablation using sampling-based estimates instead of the learned approximator.","section":"Section 4.1, Implementation Details"},{"comment":"The paper reports a single run for each configuration, and many of the headline improvements are small (e.g., MSRVTT text-to-video R@1 49.4 vs. 48.6 for HBI; MSRVTT-QA accuracy 46.4 vs. 46.2). Without standard deviations across seeds or a paired test, the claim that HBI V2 consistently surpasses existing methods is not statistically grounded. Please report variance across at least 3 seeds, or provide significance tests for the main comparisons.","section":"Section 4.2, Tables 1–3"}],"minor_comments":[{"comment":"The notation for the reconstructed representations is ambiguous; in particular, V_c^f and T_c^w appear to be sequences of repeated copies of a single cross-modal vector, and the dimensions of the MLP outputs for γ and δ are not stated. Please clarify whether γ and δ are scalar or per-token and specify the output shapes.","section":"Section 3.1, Eq. (2)–(5)"},{"comment":"The citation for 'core interaction' points to a nuclear physics paper (Jeukenne et al.), which does not appear to be a cooperative-game-theory source; please verify and replace.","section":"Section 2.2, reference [45]"},{"comment":"The exact CLIP variant (e.g., ViT-B/32 vs. ViT-L/14, input resolution) is not specified; please state it for reproducibility.","section":"Section 4.1, Implementation Details"},{"comment":"The claim that the fine-grained alignment from HBI V2 allows a 'simplified answer prediction head' is not directly supported, since no comparison with a stronger head on the same features is provided.","section":"Section 3.3, Task-Specific Prediction Heads"},{"comment":"The discussion of the γ and δ convergence curves is qualitative; the statement that the model 'adaptively reduces cross-modal information to widen the feature distribution' should be backed by quantitative analysis or a controlled experiment.","section":"Section 4.4, Fig. 9"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a substantial extension of a CVPR 2023 paper and the experimental scope is appropriate for TPAMI. The main risk is that the game-theoretic component may be decorative; the authors need to demonstrate that the Banzhaf target is computed from a valid characteristic function and that the approximation is faithful. I would not reject the paper on the current evidence, but the load-bearing issues require a careful revision and re-review. The citation problem in [45] may indicate a broader need to check references."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a decent engineering extension of the authors' own CVPR 2023 HBI paper. The genuinely new pieces are the reconstructed representation (Eq. 3 and Eq. 5) and the task-specific heads for QA and captioning; the core Banzhaf interaction and token merging come from [11]. The gains are modest—roughly 0.8 R@1 on MSRVTT retrieval and a few CIDEr points on captioning—but they are consistent across seven datasets, and the ablations show each component contributes. The self-distillation across semantic levels is a reasonable add, and the code is public.\n\nThe soft spots are real but not fatal to the empirical story. The load-bearing assumption in Section 3.2.1 is that the cross-modal similarity S can act as the characteristic function and satisfies conditions (a)-(c). The paper never verifies this, and the actual S in Eq. 12 is only defined for full video/text pairs, not for arbitrary coalitions or the merged coalition [{i,j}] that Eq. 1 requires. Since S is a weighted average of per-frame max similarities, merging a strongly matched pair does not necessarily raise the payoff, and a mismatched pair does not necessarily lower it. So the Banzhaf targets may not encode the claimed semantic correspondence. The tiny-model approximation of I is also underspecified—no details on its training data, capacity, or approximation error—and no variance or error bars are reported anywhere, so the headline gains sit within plausible run-to-run noise.\n\nThere is also a citation inconsistency: the contributions claim to be the first to introduce cooperative game theory into fine-grained video-language learning, despite the paper being an extension of the authors' own HBI and the existence of TG-VQA, which also uses game theory. That claim should be toned down.\n\nAll that said, the paper is honest about its lineage and the ablations make a reasonable case that the added losses help. The central problem is that the theoretical justification for the Banzhaf targets is asserted, not shown. I do not think the paper is incoherent, but the mechanism behind the gains is less clear than the authors argue.\n\nWho is this for? Researchers in video-language pre-training and retrieval who want a reproducible baseline with a game-theoretic flavor. It deserves a serious referee, but the referee should push for verification of the characteristic-function conditions, a proper description and validation of the approximator, and error bars on the main tables. Recommend sending to peer review with these requests.","headline":"Solid empirical extension of the authors' CVPR 2023 HBI work, but the game-theoretic framing rests on an unverified assumption about the characteristic function.","tokens_in":26805,"tokens_out":1781,"would_cite":false,"duration_ms":18407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-grained video-language alignment can be learned without manual annotations by treating frames and words as cooperative game players and using Banzhaf Interaction values as training targets.","keywords":["video-language representation learning","Banzhaf interaction","cooperative game theory","fine-grained cross-modal alignment","text-video retrieval","video question answering","video captioning","contrastive learning"],"falsifier":"Compute $S$ from Eq. 12 on real batches and check the three inequalities in Section 3.2.1 for clear positive pairs (a frame containing the object named by the word) and clear negative pairs; if a positive pair fails to make the payoff difference negative or a negative pair fails to make it positive, the premise is violated. A cheaper decisive probe is to replace the Banzhaf targets in Eq. 8 with permuted interaction values and compare downstream scores; if performance survives, the specific game-theoretic semantics are not what carries the gain.","tokens_in":25824,"feed_emoji":"🎬","tokens_out":11154,"duration_ms":92450,"temperature":0.7,"pith_summary":"Video-language models usually align whole videos to whole sentences through contrastive learning, which cannot say which frame matches which word. This paper argues that fine-grained matches can be learned without manual labels by modelling frames and words as players in a cooperative game, with the cross-modal similarity score as the payoff and the Banzhaf Interaction index as the measure of how much a frame-word coalition contributes to that payoff. The proposed HBI V2 (Hierarchical Banzhaf Interaction V2) uses these interaction values as soft training targets for a small prediction head, repeats this supervision at entity, action, and event levels by merging tokens into coalitions, and blends single-modal and cross-modal features to reduce bias in the interaction computation. If the claim is right, the same objective improves text-video retrieval, video question answering, and video captioning, and the interaction values also serve as a visualization of which words bind to which frames.","feed_headline":"Cooperative game theory sharpens video-text alignment","feed_subtitle":"Banzhaf interaction over frame-word coalitions boosts retrieval, QA, and captioning without manual frame-word labels.","key_machinery":"The load-bearing object is the Banzhaf Interaction index in Eq. 1: for a coalition $\\{i,j\\}$, it averages over all subsets $C$ of the remaining players the difference $\\phi(C \\cup [\\{i,j\\}]) + \\phi(C) - \\phi(C \\cup \\{i\\}) - \\phi(C \\cup \\{j\\})$, so a high value means the pair cooperates more than their separate contributions would suggest. The paper sets the characteristic function $\\phi$ to the cross-modal similarity $S$ of Eq. 12, a weighted average of per-frame maximum frame-word alignment scores, and it uses this index as the training target for a prediction head whose output $R$ is matched to the index by a KL-divergence loss. Around this object, the machinery consists of a representation reconstruction module that blends single-modal and cross-modal features with learnable weights (Eqs. 2-5), and a token-merge module that clusters tokens via density-peak K-nearest-neighbor search and attention, so the same interaction is computed at entity, action, and event levels.","core_discovery":"The central claim is that the uncertainty in fine-grained video-text correspondence, which frame pairs with which word, at what granularity, and with what intensity, can be handled by formulating the correspondence as a multivariate cooperative game. Video frames and text words are the players; the cross-modal similarity function is the characteristic function; and the Banzhaf Interaction of a coalition measures the coalition's incremental contribution to the total similarity score beyond what the players contribute separately. HBI V2 trains a prediction head to output this interaction index, using KL divergence to match the exact index computed from the similarity function, and because exact computation is NP-hard it learns a tiny estimator of the index for speed. The paper also claims that a reconstructed representation, formed as a learnable blend of single-modal and cross-modal encodings, reduces bias in the Banzhaf values, and that stacking token-merge modules produces entity-, action-, and event-level interactions. On three retrieval benchmarks, three video-QA benchmarks, and one captioning benchmark, the method reports consistent gains over its predecessor HBI and over task-specific methods.","pith_inferences":["A direct empirical check of the three characteristic-function assumptions would settle whether the training targets literally encode semantic correspondence: sample frame-word pairs, compute $S$ from Eq. 12, and test the inequalities in Section 3.2.1; this probe is within reach and is not reported in the paper.","The same Banzhaf supervision scheme should transfer to other paired modalities such as image-text, audio-text, or video-audio, since the mechanism only requires a tokenization and a differentiable cross-modal similarity, so a natural extension is to apply HBI-style reconstruction and hierarchy there.","The reported gains depend on the choice of pretrained vision-language backbone and its token granularity; ablating the reconstruction weights with frozen versus finetuned encoders would separate the game-theoretic signal from representation quality, which the paper does not do.","The hierarchical interaction visualization could be validated as an interpretability tool by comparing the model's frame-word interaction weights against human-annotated alignments on a small probe set, turning the qualitative figures into a quantitative benchmark."],"forward_implications":["Fine-grained frame-word alignment can be supervised without any manual annotation, because the Banzhaf index turns coarse video-text pair labels into dense soft targets.","The interaction loss is dropped at inference time, so the method's interpretability and alignment gains do not slow down deployment.","The same encoder with task-specific heads covers text-video retrieval, video question answering, and video captioning, suggesting the interaction objective is a general video-language learning signal rather than a retrieval-specific trick.","Because the exact index is NP-hard, the paper's use of a learned estimator means the method trades a small approximation error for tractability; the reported inference time on MSRVTT is only about one second more than the baseline on the test set.","The hierarchical visualization shows that coalitions of words and clips have higher semantic similarity than individual frame-word pairs, which the paper offers as evidence that the action- and event-level interactions capture coarser correspondences."],"supporting_citations":[{"why":"supplies the axiomatic Banzhaf Interaction index that the paper uses as the fine-grained training target.","marker":"[9]"},{"why":"gives the weighted Banzhaf interaction formulation cited as the interaction index concept.","marker":"[10]"},{"why":"is the predecessor HBI framework that HBI V2 extends and uses as the main internal baseline.","marker":"[11]"},{"why":"provides the pretrained vision and text encoders whose features feed the game-theoretic module.","marker":"[31]"},{"why":"supplies the density-peak K-nearest-neighbor clustering used to merge tokens into coalitions.","marker":"[52]"},{"why":"is the contrastive loss that the final training objective augments with the Banzhaf interaction loss.","marker":"[66]"},{"why":"establishes that exact power-index computation is NP-hard, motivating the learned estimator for Banzhaf values.","marker":"[75]"}],"fun_headline_variants":["Banzhaf Interaction sharpens video-text fine-grained alignment","Video-text as a game: Banzhaf Interaction wins benchmarks","Hierarchical Banzhaf Interaction refines multimodal representation learning","Banzhaf Interaction handles uncertainty in video-text correspondence","Game-theoretic Banzhaf yields gains in video retrieval, QA, captioning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the cross-modal similarity $S$ used as the characteristic function actually rewards strongly matched frame-word pairs and punishes irrelevant ones in the sense required by conditions (a)-(c); if $S$ violates those inequalities, the Banzhaf Interaction values used as training targets do not encode the claimed semantic correspondence.","fun_headline_variants_meta":{"raw":{"variants":["Banzhaf Interaction sharpens video-text fine-grained alignment","Video-text as a game: Banzhaf Interaction wins benchmarks","Hierarchical Banzhaf Interaction refines multimodal representation learning","Banzhaf Interaction handles uncertainty in video-text correspondence","Game-theoretic Banzhaf yields gains in video retrieval, QA, captioning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3127,"prompt_tokens":990,"completion_tokens":2137,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2046}},"tokens_in":606,"tokens_out":2137,"duration_ms":16442,"temperature":1.0,"reasoning_tokens":2046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:05:59.030686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $S$ from Eq. 12 on real batches and check the three inequalities in Section 3.2.1 for clear positive pairs (a frame containing the object named by the word) and clear negative pairs; if a positive pair fails to make the payoff difference negative or a negative pair fails to make it positive, the premise is violated. A cheaper decisive probe is to replace the Banzhaf targets in Eq. 8 with permuted interaction values and compare downstream scores; if performance survives, the specific game-theoretic semantics are not what carries the gain.","supporting_citations":[{"cited_title":"Learning Transferable Visual Models From Natural Language Supervision,","cited_arxiv_id":null,"evidence_quote":"provides the pretrained vision and text encoders whose features feed the game-theoretic module."},{"cited_title":"Np-completeness for calculating power IEEE TRANSACTIONS ON PATTERN ANAL YSIS AND MACHINE INTELLIGENCE, VOL. XX,NO. XX, XXX. XXXX 15 indices of weighted majority games,","cited_arxiv_id":null,"evidence_quote":"establishes that exact power-index computation is NP-hard, motivating the learned estimator for Banzhaf values."}],"review_version":1}