{"id":"59792c0c-e5a1-4aa3-a16a-359c040e7b41","arxiv_id":"2608.03112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A training-free two-stage token pruning method for video-language models, using eigenvalue decay of token correlations to set a content-adaptive retention ratio.","lead":"This paper presents a way to make video understanding AI faster by removing redundant frames and image patches before they reach the language model. It decides how much to prune separately for each video, based on how repetitive the video's visual content is.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive retention ratio is calibrated per dataset and ratio on the same test benchmarks, so the reported gains may reflect per-dataset tuning of κ rather than the content-adaptive mechanism.","rationale":"The reader's weakest assumption concerned whether the exponential decay rate α faithfully measures task-relevant redundancy. That is a legitimate scientific concern. However, the more directly falsifiable and load-bearing issue is that the adaptive mechanism is only tested after per-dataset calibration of κ on the evaluation sets themselves (Table 6), making it impossible to separate adaptive benefit from test-set tuning. This does not accuse the authors of misconduct; it is a methodological circularity in the reporting protocol. The two-stage pruning component has internal support (Figure 4, Table 4) and the adaptive component shows a consistent internal ablation, so a REJECT would be too harsh. But the central claim about content-adaptivity requires an out-of-sample evaluation before full acceptance. Hence CONDITIONAL, with the specific validation experiment as the acceptance criterion.","tokens_in":12314,"tokens_out":1295,"duration_ms":16291,"concrete_test":"Split each benchmark into a validation set and a test set. Tune κ once on the validation set (or on one dataset, then transfer it without retuning), freeze κ, and evaluate on the held-out test sets. If the reported gains shrink or vanish under this protocol, the headline claim of content-adaptive superiority is not supported. Additionally, report the per-video distribution of β and compare against a fixed-ratio two-stage baseline whose ratio equals the average β over the test videos.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the adaptive rule β = κ/α generalizes across videos and outperforms fixed-ratio baselines at equal average budgets. However, the protocol sets κ separately for each dataset, model, and target retention ratio (Table 6), and these κ values must be chosen by looking at results on the very benchmarks where gains are reported. The exponential functional form in Eq. 1 is likewise selected via the Section 4.4.2 ablation on VideoDC. Consequently, the comparison is not out-of-sample: the 'adaptive' component is effectively fitted to the test sets. The paper reports no held-out validation and no error bars, so a reader cannot determine whether the gains come from (a) the per-video variation in β driven by the eigenvalue decay rate α, or (b) the per-dataset/per-ratio tuning of the constant κ. The internal ablation in Table 4 shows that adaptive β helps over fixed-ratio two-stage pruning at the chosen κ, but this does not disentangle calibration from genuine adaptivity; a fixed-ratio method with κ tuned per dataset might show similar gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, two-stage visual token pruning method for video-language models. The first stage selects diverse frames via diversity-based pruning, and the second stage prunes tokens within retained frames using a content-adaptive retention ratio beta = kappa / alpha, where alpha is the exponential decay rate fitted to the top eigenvalues of the token-embedding correlation matrix. The method is evaluated on LLaVA-Video, InternVL3, and Qwen2.5VL across five video benchmarks, comparing against fixed-ratio training-free baselines such as DivPrune, PruMerge, FastV, AvgPool, and LLaVA-Scissor. The reported results show consistent accuracy improvements, with the largest gains on video captioning (VideoDC), e.g., about 6-7% absolute accuracy at 10% retention, alongside large TFLOP reductions.","tokens_in":12572,"tokens_out":2838,"duration_ms":29371,"significance":"If the evaluation protocol were fully out-of-sample, this would be a practical and valuable plug-in efficiency method: it is post-hoc, requires no training, and the two-stage frame-then-token design is well motivated by temporal and spatial redundancy. The paper also provides useful ablations separating the two-stage mechanism from the adaptive ratio. However, the significance is conditional on the calibration protocol: the adaptive component depends on a constant kappa that is tuned per dataset, model, and target retention ratio, and the reported gains may reflect this per-benchmark tuning rather than a content-adaptive mechanism that generalizes. The absence of error bars and the selection of the exponential functional form on the test set further weaken the strength of the empirical claim.","major_comments":[{"comment":"The central claim that the adaptive rule beta = kappa / alpha generalizes across videos is not supported by the current evaluation protocol. Table 6 shows that kappa is set separately for each dataset, each model, and each target retention ratio, and the reported accuracies are on the same test benchmarks used to select these values. Since kappa directly scales the retention ratio, this is effectively per-benchmark calibration of the average budget. The comparison to fixed-ratio baselines is therefore not out-of-sample. The paper should provide a held-out validation protocol, e.g., tuning kappa on a validation split or on one dataset and transferring to the others, and report the sensitivity of results to the choice of kappa. Without this, the reader cannot distinguish gains from the per-video variation in alpha from gains due to per-dataset tuning.","section":"§4.1 and Table 6"},{"comment":"The exponential functional form in Eq. (1) and the choice K=32 are justified only through a test-set ablation on VideoDC (Table 5). Formula selection on the same benchmark where the final numbers are reported inflates the apparent benefit of the exponential fit. Additionally, the paper asserts, but does not demonstrate, that the eigenvalue decay rate alpha faithfully measures how many tokens the model needs. A direct test would be to compare the per-video beta against a per-video oracle retention ratio, or at least to show that the chosen functional form and K transfer to held-out datasets without re-fitting. As written, the 'principled' derivation is an empirical fit with a hand-set constant.","section":"§3.2 and §4.4.2"},{"comment":"The claim of consistent improvement over baselines is not supported by any measure of uncertainty. No error bars, repeated seeds, or significance tests are reported anywhere. At 30% retention, the gains over DivPrune are very small on multiple-choice datasets (Next-QA: 82.15 vs 82.05; Video-MME: 60.52 vs 60.48 in Table 1), so the strong statement in §4.1 that 'our method consistently outperforms the baseline' across all retention ratios would benefit from statistical support or a more nuanced phrasing. The large gains on VideoDC and VideoChatGPT at low retention are encouraging, but without uncertainty quantification it is unclear which of the reported differences are reliable.","section":"§4.1 and Tables 1-3"}],"minor_comments":[{"comment":"The last two columns (time and TFLOPs) appear to list values only on the DivPrune rows; the rows for 'Ours' show time but no explicit TFLOP value, making the claimed ~95.5% TFLOP reduction difficult to verify directly from the table. Please format the table so that each method row has its own time and TFLOP entry.","section":"Table 1"},{"comment":"There is a typo in the first paragraph of Section 4: 'conduced' should be 'conducted'. Also, 'LLaV A-Video' is written with inconsistent spacing throughout; please standardize.","section":"§4"},{"comment":"The statement that 'the top 32 eigenvalues capture most of the spectral energy' is presented without supporting evidence or a reference; please provide a quantitative justification or cite prior work.","section":"§3.2"},{"comment":"The paper does not include a limitations section; given the calibration sensitivity identified above, a brief discussion of limitations (e.g., sensitivity to kappa, lack of uncertainty estimates, evaluation on specific VLM families) would improve the manuscript.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper is a solid, well-scoped efficiency study. The new bit is using the eigenvalue decay rate of the token correlation matrix to set a per-video token retention ratio, on top of a two-stage frame-then-token pruning scheme. That is a sensible idea, and the authors show it consistently beats fixed-ratio training-free baselines across three VLMs and five datasets, which is more breadth than most pruning papers.\n\nWhat I like: the two-stage framing is natural for video; the eigenvalue-based redundancy estimate is cheap (top-32 via power iteration); the ablations separating the two-stage component from the adaptive component are useful; and they include runtime/TFLOP numbers, not just accuracy. The gains at 10% retention on captioning are meaningful if they hold up.\n\nWhere I'd push: the biggest soft spot is the protocol for κ. Table 6 shows κ is set per dataset, per model, and per target retention. If κ is tuned on the same test benchmarks used for reporting, then the adaptive component is not really out-of-sample. The paper says κ is set so the average retention matches the baseline, but they never report the achieved average β or its distribution. Without that, a reader can't rule out that the gains come from a favorable average mismatch rather than from per-video adaptation. This is not fatal—the fixed-ratio baseline is also at the same nominal average, so there is some signal—but it needs a validation-set calibration and error bars. The choice of exponential fit is also ablated on VideoDC, one of the test sets, which slightly inflates the result.\n\nMinor: no error bars or repeated seeds anywhere, and some of the wins are tiny (0.1 point on NextQA, 0.03 on Video-MME). But the consistent direction across many conditions is reassuring.\n\nVerdict: worth a serious referee. I'd ask for a holdout-based κ selection, reporting of actual average β, and at least a handful of seeds. The core idea is simple and probably right; the paper just needs to demonstrate that the adaptivity is doing the work, not the calibration.","headline":"A useful pruning method whose adaptive ratio is plausible but under-validated: the gains could partly come from per-dataset calibration of κ rather than from the content-adaptive mechanism.","tokens_in":13063,"tokens_out":3842,"would_cite":true,"duration_ms":38878,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that a two-stage, content-adaptive token-pruning rule outperforms fixed-ratio baselines for video-language models, delivering over 6% accuracy gains on a captioning benchmark at 10% token retention.","keywords":["visual token pruning","video-language models","adaptive retention ratio","eigenvalue decay","training-free inference acceleration","temporal redundancy","frame-level pruning","token correlation matrix"],"falsifier":"Construct or find a video that is mostly static but contains one short, task-critical event, compute its $\\alpha$, prune at the resulting $\\beta$, and compare accuracy against a fixed-ratio baseline at the same average budget; if the adaptive rule loses on the critical detail, the decay rate is not faithfully measuring needed tokens.","tokens_in":12138,"feed_emoji":"✂️","tokens_out":5676,"duration_ms":51109,"temperature":0.7,"pith_summary":"This paper tries to establish that video-language models can be made much cheaper to run by pruning visual tokens in two stages, first dropping redundant frames and then dropping redundant tokens within the kept frames, with the number of tokens kept decided per video rather than fixed in advance. The authors argue that existing training-free pruning methods use a one-size-fits-all retention ratio and ignore temporal redundancy, which wastes computation and loses accuracy. They report consistent gains over fixed-ratio baselines across three vision-language models and five video benchmarks, including over 6% higher accuracy on a video captioning benchmark at 10% token retention, while cutting LLM computation by roughly 95.5%. If correct, the recipe is a plug-in efficiency improvement that needs no fine-tuning.","feed_headline":"Adaptive pruning beats fixed-ratio token cuts in video LLMs","feed_subtitle":"Two-stage frame-and-token pruning sets each video's budget from its own redundancy, lifting accuracy even at 10% retention.","key_machinery":"The load-bearing object is the exponential decay rate $\\alpha$ of the eigenvalue spectrum of the token-correlation matrix: after pruning frames, the method computes pairwise correlations among the surviving token embeddings, extracts the top 32 eigenvalues via low-rank SVD, and fits $\\log \\lambda_n = \\log A - \\alpha n$. A steep spectrum (large $\\alpha$) signals high redundancy and yields a small retention ratio $\\beta=\\kappa/\\alpha$; a flat spectrum signals diversity and yields a larger budget. The first stage, frame-level diversity selection at ratio $\\gamma$, handles temporal redundancy, and the second stage prunes tokens within kept frames. The method is post-hoc, requiring no training, and the SVD is deliberately low-rank to keep the overhead small.","core_discovery":"The central discovery, stated in the paper's terms, is that the right number of visual tokens for a video can be read off the correlation structure of its token embeddings. After frame-level diversity selection, the method builds the correlation matrix of the remaining token embeddings, takes its top eigenvalues, and fits an exponential curve to their decay; the decay rate $\\alpha$ is treated as a redundancy score, and the token-level retention ratio is set to $\\beta = \\kappa/\\alpha$, with $\\kappa$ chosen so the average budget matches the baselines. The paper reports that at equal average retention (30%, 15%, 10%) this adaptive rule beats fixed-ratio training-free baselines on VideoDC, VideoChatGPT, NextQA, PerceptionTest, and Video-MME, with the largest gains on open-ended captioning and QA tasks. It also reports the same pattern across InternVL3 and Qwen2.5VL and across 1B, 2B, and 7B model sizes.","pith_inferences":["Beyond the paper, the same decay rate could be reused as a general redundancy signal, e.g. to choose how many frames to keep in the first stage rather than fixing $\\gamma=0.5$, which the paper tunes per model.","Beyond the paper, the correlation-based budget could be combined with other token-selection rules, since it only decides how many tokens to keep and is agnostic to which selector picks them.","Beyond the paper, a testable extension is to apply the adaptive ratio to images by taking a single frame; the paper evaluates videos only, but the correlation argument is not video-specific.","Beyond the paper, one could monitor $\\alpha$ as a video progresses and re-prune dynamically at the clip level, since the eigenvalue estimate is cheap enough to recompute."],"forward_implications":["Plug-in efficiency: any off-the-shelf video-language model can run the same two-stage pruning at inference time without retraining, so latency reductions apply to already-deployed models.","Content-dependent budgets: videos with repetitive, static content automatically receive fewer tokens than dynamic ones, so the same global compute budget is spent where it matters.","Large compute savings at low retention: at 10% average retention the method cuts LLM TFLOPs by about 95.5% while improving accuracy over fixed-ratio baselines on captioning and open-ended QA.","Consistency across model families and sizes: the gains appear on LLaVA-Video, InternVL3, and Qwen2.5VL, and on 1B, 2B, and 7B parameter sizes, suggesting the rule transfers rather than being tuned to one model.","Two-stage ordering matters: the paper's ablation shows that frame-level pruning followed by token-level pruning outperforms token-level pruning alone at the same total budget."],"supporting_citations":[{"why":"Supplies the diversity-based token selection used in both pruning stages and the strongest fixed-ratio baseline.","marker":"[1]"},{"why":"Training-free baseline that also prunes across frames and tokens; the main content-adaptive competitor.","marker":"[24]"},{"why":"Attention/merge-based training-free baseline compared at every retention ratio.","marker":"[20]"},{"why":"Attention-based training-free baseline compared at every retention ratio.","marker":"[4]"},{"why":"Provides the AvgPool training-free baseline used in the comparison.","marker":"[25]"},{"why":"The 7B video-language model on which the main experiments and ablations run.","marker":"[30]"},{"why":"Second vision-language model used to test whether the gains transfer.","marker":"[32]"},{"why":"Third vision-language model used to test transfer.","marker":"[2]"},{"why":"Captioning dataset where the largest accuracy gains are reported.","marker":"[16]"},{"why":"Open-ended QA dataset used to measure quality of generated answers.","marker":"[18]"}],"fun_headline_variants":["Video LLMs slash 95% compute with adaptive token pruning","Per-video token budgets cut LLM compute 95%","Adaptive frame and token pruning speeds video LLMs","Video LLM pruning sets token ratio from content, no training","Efficient video LLM inference via content-aware token pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the steepness of the eigenvalue drop-off in a video's token correlations tells how many tokens the model truly needs, so a video whose tokens look redundant but whose few distinctive details matter would be pruned too hard.","fun_headline_variants_meta":{"raw":{"variants":["Video LLMs slash 95% compute with adaptive token pruning","Per-video token budgets cut LLM compute 95%","Adaptive frame and token pruning speeds video LLMs","Video LLM pruning sets token ratio from content, no training","Efficient video LLM inference via content-aware token pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000745,"raw_usage":{"total_tokens":3353,"prompt_tokens":1009,"completion_tokens":2344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2261}},"tokens_in":625,"tokens_out":2344,"duration_ms":17282,"temperature":1.0,"reasoning_tokens":2261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:52:24.815007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or find a video that is mostly static but contains one short, task-critical event, compute its $\\alpha$, prune at the resulting $\\beta$, and compare accuracy against a fixed-ratio baseline at the same average budget; if the adaptive rule loses on the critical detail, the decay rate is not faithfully measuring needed tokens.","supporting_citations":[{"cited_title":"Divprune: Diversity-based visual token pruning for large multimodal models","cited_arxiv_id":null,"evidence_quote":"Supplies the diversity-based token selection used in both pruning stages and the strongest fixed-ratio baseline."},{"cited_title":"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models","cited_arxiv_id":null,"evidence_quote":"Attention-based training-free baseline compared at every retention ratio."},{"cited_title":"Video detail caption","cited_arxiv_id":null,"evidence_quote":"Captioning dataset where the largest accuracy gains are reported."}],"review_version":1}