{"id":"84d64c06-3fc3-4b23-9462-d5704d5e2846","arxiv_id":"2411.12593","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AdaCM2 adaptively evicts low-attention video tokens using cross-modality attention scores, keeping memory bounded while improving accuracy on long-video QA, captioning, and classification.","lead":"This paper introduces AdaCM2, a method that reduces memory use for very long videos by keeping the visual tokens most relevant to the text question, layer by layer. If it works, video-understanding models could process hours of footage without running out of memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that cross-modality attention scores identify answer-critical tokens is under-supported: the only behavioral evidence is a random-eviction comparison, not a no-eviction upper bound, so the 4.5% gain could come from other components.","rationale":"I read the paper as making two linked claims: (1) cross-modality attention scores identify the visual tokens most relevant to a text query, and (2) evicting low-scoring tokens therefore preserves accuracy while bounding memory. The memory bound in Eq. 8 is internally consistent, and the random-eviction ablation in Figure 7 shows that attention-based selection is better than random selection at the same compression level. That is real evidence, but it is not sufficient for the central claim. Random selection is a weak baseline: even a flawed importance signal can beat random, and the reported 4.5% LVU improvement over MA-LMM could plausibly come from the regressive frame-by-frame query construction or from fine-tuning Q-Former on the target datasets, neither of which is independently ablated. The missing control is a no-eviction (beta = 1) run, which would establish the accuracy cost of the memory reduction and, by comparison with the random baseline, isolate the value of the cross-modality ranking. This is a testable and concrete gap, not a fatal flaw. The paper otherwise has a clear mechanism, a plausible geometric memory argument, and reasonable experimental breadth across LVU, Breakfast, COIN, MSRVTT-QA, MSVD-QA, and captioning sets. The reader's CONDITIONAL verdict already captures the need for stronger validation; my concern identifies a specific missing control rather than moving the verdict. I therefore leave the verdict unchanged, with the concrete test above as the natural next step for the authors.","tokens_in":13980,"tokens_out":7314,"duration_ms":76689,"concrete_test":"Run a no-eviction control on a memory-feasible task: on MSVD-QA and MSRVTT-QA, train/evaluate AdaCM2 with beta = 1 (no eviction) and with the proposed beta = 0.1 (same alpha), plus random eviction at the same budget. If beta = 1 accuracy is materially higher than beta = 0.1, the eviction rule loses answer-critical tokens and the claimed cross-modality advantage needs re-scoping; if beta = 0.1 matches beta = 1 and beats random, the concern is resolved. For LVU, run the same control on a short-video subset (e.g., <=100 frames) where beta = 1 fits in memory, and report results over at least three seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the cross-modality attention score in Eq. 5, used by the eviction rule in Eq. 7, ranks visual tokens by their contribution to the correct answer. The paper's only behavioral evidence for this is Figure 7, which compares AdaCM2's attention-based eviction to random eviction on LVU. Random eviction is not an upper bound; a selection rule can beat random while still discarding answer-critical tokens. Consequently, the 4.5% LVU gain over MA-LMM (Table 1) cannot be attributed to the cross-modality ranking. It could be driven by the regressive Q-Former query, the fine-tuned Q-Former parameters, or the frame-by-frame cache design, none of which is ablated separately. The ablation also lacks a no-eviction control (beta = 1) that would show how much accuracy is actually sacrificed by the memory reduction itself. Without that control, the central claim that cross-modal attention preserves answer-relevant information is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaCM^2, a video-language model that reduces GPU memory for long-video understanding by selecting which visual tokens to keep in a growing video cache using cross-modality attention scores from a Q-Former. The visual encoder and LLM are frozen; only Q-Former parameters are fine-tuned. The method processes frames regressively, partitions the cache into recent and previous parts, and evicts previous-cache tokens with low accumulated cross-modality attention scores. The authors report state-of-the-art or competitive results on LVU, Breakfast, COIN, MSRVTT-QA, MSVD-QA, and captioning benchmarks, and claim up to 65% GPU memory reduction with near-constant memory as video length grows. The central derivation, Eq. (8), gives a constant asymptotic cache size under a fixed per-step reduction ratio.","tokens_in":14352,"tokens_out":4489,"duration_ms":48569,"significance":"If the claims hold, the contribution is practically significant: it addresses a real bottleneck in LLM-based video understanding, where memory grows with frame count. The paper has several genuine strengths: the memory bound in Eq. (8) is a correct mathematical consequence of the reduction algorithm; the random-eviction comparison in Figure 7 is a useful sanity check for the selection signal; and the plug-and-play framing around InstructBLIP is attractive. The method is also evaluated across multiple datasets and tasks. However, the central mechanistic claim — that cross-modality attention scores preserve answer-critical tokens — is under-supported: the evidence against random eviction is not an upper-bound control, there are no error bars or run counts, and the adaptive layer-wise component is not actually exercised in the experiments because alpha and beta are fixed to 0.1 for all layers. The significance will be much clearer once these missing controls are supplied.","major_comments":[{"comment":"The paper motivates a layer-wise adaptive reduction from Observation 2 (Section 3.2), but Eq. (8) derives the memory bound using a single per-step ratio r = alpha + (1-alpha)beta, and Section 5.4 states that alpha and beta are set to 0.1 for all layers. A fixed alpha=beta=0.1 means the reduction is uniform across layers, so the central 'adaptive layer-wise' claim is not actually exercised in any experiment. Please either generalize Eq. (8) to layer-dependent r_l (with an appropriate product/sum bound) and report experiments with distinct per-layer ratios, or soften the layer-wise claim to match the evaluated configuration.","section":"§4.2 / Eq. (8), §5.4"},{"comment":"The only behavioral evidence that cross-modality attention scores identify answer-critical tokens is a comparison with random eviction. Beating random eviction does not establish that the retained tokens are the answer-critical ones; a selection rule can beat random while still discarding important information. The paper should add a no-eviction (or beta=1) control on the same LVU tasks to quantify how much accuracy is sacrificed by the memory reduction itself, and ideally ablate the regressive Q-Former query and the cache-partition design separately. Without these controls, the 4.5% LVU improvement over MA-LMM cannot be attributed specifically to the cross-modality ranking.","section":"§5.4 / Figure 7"},{"comment":"All accuracy numbers are reported as single values without the number of runs, standard deviations, or significance tests. Several differences against the strongest baseline are small (e.g., COIN 93.3 vs. 93.2, MSRVTT METEOR 33.0 vs. 33.4, YouCook2 CIDEr 125.6 vs. 131.2), and these could easily be within run-to-run variation for fine-tuned multimodal models. Please report at least three runs with mean and standard deviation, and state whether the reported numbers are averages or best runs.","section":"Tables 1–3"},{"comment":"The abstract's 'up to 65% GPU memory reduction' claim is not supported by any table or numerical protocol. Figure 6 shows qualitative curves, but the paper does not report the measured memory values, the batch size, the number of frames at which the 65% figure is computed, or how the memory footprint is measured (e.g., peak CUDA memory vs. model plus cache). Please add a table with memory measurements and the exact measurement conditions, and state whether the 65% reduction is relative to MA-LMM, InstructBLIP, or another baseline.","section":"Abstract / §5.3 / Figure 6"},{"comment":"Equation (5) defines the cross-modality attention score S_c^t(i) by summing over j = 1..N, where N is the number of query and text tokens. The surrounding text says the score measures correlation with 'text tokens,' but the sum includes the learnable Q-Former query tokens, which are trained parameters, not text. This conflation matters because the eviction rule in Eq. (7) is driven by S_c^t. Please either restrict the sum to the text-token subset, or explicitly justify why query-token attention belongs in the scoring signal.","section":"§4.2 / Eq. (5)"}],"minor_comments":[{"comment":"There are multiple typos and inconsistencies: 'weight matrics' (Section 4.2), 'MSVT' instead of 'MSRVTT' (Section 5.4), inconsistent 'AdaCM 2' / 'AdaCM2' spacing, and the Table 1 title 'The underlinednumber means the second best.' A careful proofread is needed.","section":"Throughout"},{"comment":"Figure 2 is presented as a qualitative zero-shot case study on Ego4D, but the paper does not state whether this video is part of any quantitative benchmark or how it was selected. A brief sentence clarifying its role would prevent confusion.","section":"Figure 2"},{"comment":"Observation 2 is supported by a single MSR-VTT video. The claim that redundancy varies across layers would be strengthened by reporting statistics over multiple videos and datasets, with error bars or box plots.","section":"Figure 3c"},{"comment":"The caption and text claim state-of-the-art performance, but mPLUG-2 achieves higher METEOR and CIDEr on MSRVTT. The dismissal of mPLUG-2 due to pretraining overhead should be made in the text as a qualifying statement rather than in the table caption only.","section":"Table 3"},{"comment":"The loss in Eq. (9) uses symbols V and P that are not defined in the text generation section; please define all notation, including the role of the video features V in a text-generation loss.","section":"§4.3 / Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper's advertised headline results (4.5% accuracy gain on LVU, 65% memory reduction) are not yet backed by sufficient experimental control: no error bars, no no-eviction upper bound, and no tabulated memory measurements. The missing experiments are feasible and within the manuscript's scope, so I would not recommend rejection. I also note that the 'adaptive layer-wise' claim is significantly stronger than what the fixed alpha=beta=0.1 configuration actually evaluates; the authors should either implement per-layer ratios or reframe the contribution. The strongest part of the paper is the clean geometric memory bound in Eq. (8), but the empirical attribution of the accuracy gain to the cross-modality attention ranking remains the weakest link."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a serious, readable attempt to make long-video LLMs practical. It transfers KV-cache eviction ideas into the cross-modal Q-Former, using text-conditioned attention scores to decide which visual tokens to drop. The memory bound in Eq. 8 is correct, and the empirical memory curve is nearly flat. That part is real.\n\nWhat's new: the cross-modality attention signal for token selection, and the regressive frame-by-frame Q-Former updates. Both are reasonable and give the method a fresh feel. The LVU numbers are strong (avg 67.5 vs MA-LMM's 63.0), and the gains on director, genre, and writer are large. The random-eviction ablation is the right first check, and it shows the attention ranking beats random.\n\nThe big soft spot is attribution. The ablation compares attention-based eviction against random eviction, but random eviction is not the right upper-bound control. Without a no-eviction (beta=1) run, you cannot tell how much accuracy the reduction itself costs, and without ablating the regressive Q-Former and the fine-tuned parameters separately, the 4.5% gain over MA-LMM could come from other components. The paper also says alpha and beta are layer-wise, then fixes both to 0.1 for all layers, which contradicts the dynamic selling point. There are no error bars or multiple runs, and the 65% memory reduction is only in a figure, not a table. Code is not released.\n\nThese are fixable. The memory mechanism itself is plausible and the math checks out. The claim that cross-modal attention reliably ranks answer-critical tokens is only weakly evidenced — one heatmap plus the random comparison — but that is a standard burden for this kind of method, and the random ablation does show signal.\n\nThis paper is for people building efficient long-video video-language models. It deserves a serious referee; my recommendation is to send it to review with a request for the missing no-eviction control, per-task breakdown, and error bars. The core idea will survive; the quantitative claims need to be pinned down.","headline":"A plausible cross-modal KV-eviction method with a real memory bound, but the key ablation lacks a no-eviction control, so the 4.5% accuracy attribution is underdetermined.","tokens_in":14698,"tokens_out":2066,"would_cite":true,"duration_ms":21388,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaCM2 keeps visual tokens that the text query pays attention to and discards the rest, so long videos can be processed with bounded memory and without losing—often improving—accuracy.","keywords":["long-term video understanding","cross-modal attention","memory reduction","token eviction","video question answering","video captioning","vision-language models","video cache compression"],"falsifier":"Take a long-video VQA item whose answer depends on a small or background visual detail, e.g., a license plate or a brief object near the frame edge, run AdaCM2 with its default $\\alpha=0.1$ and $\\beta=0.1$, and check whether the correct answer survives after eviction. A systematic version would replace the cross-attention ranker with random eviction at matched token budgets: if random eviction ever matches or beats the attention-based rule on a task, that task provides a counterexample to the assumption that the attention score is a dependable importance signal.","tokens_in":13806,"feed_emoji":"🎬","tokens_out":7546,"duration_ms":68677,"temperature":0.7,"pith_summary":"AdaCM2 is a memory-reduction method that lets LLM-based video models watch very long videos by keeping only the visual tokens that the text prompt actually attends to. At each layer of the visual-textual alignment stage, it splits the running video cache into a recent part, which is always kept, and an older part, from which only the tokens with the highest cross-modality attention scores survive. The authors show this gives a bounded cache size no matter how many frames arrive, and report that it improves average accuracy by 4.5% on the seven-task LVU benchmark while reducing GPU memory by up to 65% compared with the prior MA-LMM system. If the claim holds, memory cost no longer scales with video duration, so hour-long and even unbounded video streams become practical for question answering, captioning, and classification.","feed_headline":"AdaCM2 cuts long-video GPU memory 65%, lifts accuracy 4.5%","feed_subtitle":"Cross-modal attention keeps only query-relevant visual tokens, so LLM video models can watch hours of video with near-flat memory.","key_machinery":"The load-bearing object is the cross-modality attention score $S_t^c(i)=\\sum_j S_t(j,i)$, the total attention that text/query tokens give to a visual token in a Q-Former layer. AdaCM2 treats this scalar as an importance rank for eviction: the video cache is partitioned into a recent cache and a previous cache, the recent cache is kept intact to preserve temporal continuity, and the previous cache is trimmed to its top-$\\beta$ scoring tokens. Because the trimming ratios $\\alpha$ and $\\beta$ are set per layer, the method adapts to the observation that deep layers are more redundant than shallow ones. Regressive frame-by-frame query updates make the learnable query tokens carry temporal context, and the final length-limited query vector is what the frozen LLM decodes from.","core_discovery":"The paper's central claim is that redundancy in video-text alignment is query-dependent and layer-dependent, and that this redundancy can be exploited without losing accuracy. By inspecting the Q-Former's cross-attention between text tokens and visual tokens, AdaCM2 observes that only a small subset of visual tokens in a frame receive high attention from the text, and that adjacent frames' attention patterns are highly similar, with similarity increasing in deeper layers. It therefore scores every visual token by the sum of cross-attention it receives from text tokens (Eq. 5), always keeps the most recent $\\alpha$ fraction of the cache, and from the older $(1-\\alpha)$ fraction keeps only the top-$\\beta$ scoring tokens (Eq. 7). The resulting cache size follows a geometric bound, converging to $P r/(1-r)$ with $r=\\alpha+(1-\\alpha)\\beta$ as video length grows. On this basis the paper claims 4.5% average accuracy improvement over the strongest prior memory-bank method on LVU, gains on Breakfast/COIN and on MSVD captioning, and up to 65% lower GPU memory with near-flat growth as frames increase.","pith_inferences":["The same query-guided eviction rule could be applied to other long-context modalities, such as audio or sensor streams, wherever a cross-attention score is available between a query and a per-timestep memory.","A testable prediction follows: queries about small or background details (e.g., a number on a jersey, an object behind the main action) should show a larger accuracy drop than queries about salient foreground content, because such details are more likely to live in low-attention tokens that get evicted.","The bounded-cache result suggests an online or streaming deployment where each frame is consumed once and expired tokens never need to be revisited; that would make AdaCM2 a candidate for real-time long-term video assistants, though the paper does not evaluate latency or streaming.","Layer-wise $\\alpha$ and $\\beta$ could be made question-adaptive by inspecting the entropy of the attention distribution per layer, rather than fixed global values; the paper's hyperparameter sweeps show a mid-range optimum but do not explore per-video adaptation."],"forward_implications":["With $T\\to\\infty$, the video cache converges to the constant size $P r/(1-r)$, so videos of any length can in principle be streamed through the model with bounded memory.","On the LVU benchmark the method raises average Top-1 accuracy from 63.0 to 67.5 over the prior MA-LMM baseline, with the largest gains on scene, writer, and year prediction.","Because the visual encoder and LLM stay frozen and only the Q-Former is fine-tuned, the reduction module can be dropped into existing BLIP-based video-language pipelines without retraining the large backbones.","The layer-wise adaptive ratios mean the model spends more memory where information is denser and less where frames are redundant, making memory use track the video's content rather than its duration."],"supporting_citations":[{"why":"BLIP-2's Q-Former is the architecture whose cross-attention AdaCM2 repurposes for token scoring and eviction.","marker":"[23]"},{"why":"InstructBLIP supplies the pre-trained Q-Former weights that initialize the trainable component.","marker":"[11]"},{"why":"MA-LMM is the memory-bank long-video baseline whose LVU, Breakfast, COIN, and captioning numbers AdaCM2 compares against and improves on.","marker":"[16]"},{"why":"Adaptive KV cache compression in LLMs is the direct motivation for using attention scores to decide which tokens to discard.","marker":"[14]"},{"why":"Heavy-hitter token retention establishes that a small subset of high-attention tokens can carry most of an LLM's performance, the premise AdaCM2 extends to video-text attention.","marker":"[54]"},{"why":"The LVU benchmark defines the seven long-form tasks used for the headline 4.5% accuracy gain.","marker":"[45]"}],"fun_headline_variants":["AdaCM2: 65% less GPU memory, 4.5% better video understanding","AdaCM2: Query-aware token pruning cuts video memory 65%","AdaCM2 keeps long-video memory flat, prunes tokens by query","AdaCM2: 65% less memory, 4.5% better accuracy on long videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a visual token's total cross-attention score from the text query correctly identifies which visual details are needed to answer that query, in every layer and every video; if the needed information sits in a low-attention background token, the eviction rule could delete the evidence needed for the right answer.","fun_headline_variants_meta":{"raw":{"variants":["AdaCM2: 65% less GPU memory, 4.5% better video understanding","AdaCM2: Query-aware token pruning cuts video memory 65%","AdaCM2 keeps long-video memory flat, prunes tokens by query","AdaCM2: 65% less memory, 4.5% better accuracy on long videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":4071,"prompt_tokens":1009,"completion_tokens":3062,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2971}},"tokens_in":625,"tokens_out":3062,"duration_ms":22465,"temperature":1.0,"reasoning_tokens":2971,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:14:20.825573+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a long-video VQA item whose answer depends on a small or background visual detail, e.g., a license plate or a brief object near the frame edge, run AdaCM2 with its default $\\alpha=0.1$ and $\\beta=0.1$, and check whether the correct answer survives after eviction. A systematic version would replace the cross-attention ranker with random eviction at matched token budgets: if random eviction ever matches or beats the attention-based rule on a task, that task provides a counterexample to the assumption that the attention score is a dependable importance signal.","supporting_citations":[{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023","cited_arxiv_id":null,"evidence_quote":"BLIP-2's Q-Former is the architecture whose cross-attention AdaCM2 repurposes for token scoring and eviction."},{"cited_title":"Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"InstructBLIP supplies the pre-trained Q-Former weights that initialize the trainable component."},{"cited_title":"Ma-lmm: Memory-augmented large multimodal model for long-term video understanding","cited_arxiv_id":null,"evidence_quote":"MA-LMM is the memory-bank long-video baseline whose LVU, Breakfast, COIN, and captioning numbers AdaCM2 compares against and improves on."},{"cited_title":"Model tells you what to dis- card: Adaptive KV cache compression for LLMs","cited_arxiv_id":null,"evidence_quote":"Adaptive KV cache compression in LLMs is the direct motivation for using attention scores to decide which tokens to discard."},{"cited_title":"H2o: Heavy-hitter oracle for efficient generative inference of large language models","cited_arxiv_id":null,"evidence_quote":"Heavy-hitter token retention establishes that a small subset of high-attention tokens can carry most of an LLM's performance, the premise AdaCM2 extends to video-text attention."},{"cited_title":"Towards Long- Form Video Understanding","cited_arxiv_id":null,"evidence_quote":"The LVU benchmark defines the seven long-form tasks used for the headline 4.5% accuracy gain."}],"review_version":1}