{"id":"73a2a013-eca3-4aa0-8d65-86c2c8932ffd","arxiv_id":"2411.14401","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DYTO combines hierarchical frame clustering with bipartite token merging to compress video inputs for frozen image-language models, yielding modest, inconsistent gains over training-free baselines.","lead":"DYTO is a training-free method that selects key video frames by clustering and then merges similar visual tokens, letting a frozen image-language model answer video questions. It posts small gains over training-free baselines on most benchmarks, but the paper's claim that it beats all fine-tuned models is contradicted by its own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported DYTO accuracies conflict across Table 1 and Table 3 (VideoMME 41.2 vs 42.7 for DYTO-7B; IntentQA 67.5 vs 67.3 for DYTO-34B), so the empirical basis for the claimed SOTA is internally inconsistent and needs reconciliation.","rationale":"The reader's conditional verdict is appropriate, but the weakest assumption they highlighted (CLS-token clustering and the 'second-largest' rule) is not the most load-bearing concern. The paper's own Table 3 shows that even without clustering, DYTO outperforms SlowFast on most benchmarks, so a failure of the clustering assumption would weaken but not falsify the central claim. Instead, the most serious problem is the internal inconsistency of the reported accuracy numbers: the same configuration appears with different scores in Table 1 versus Table 3. This is a concrete, verifiable flaw that undermines the reliability of all head-to-head comparisons. If the numbers are not reproducible within the paper, the empirical support for 'superior performance compared to both fine-tuned and training-free methods' collapses. My verdict remains CONDITIONAL because the method is plausible and the flaw may be correctable; the authors must reconcile the tables, provide reproducible evaluation code, and report variance or confidence intervals for the key comparisons. The reader's concern about clustering is still worth addressing, but it is not the single deciding factor.","tokens_in":14688,"tokens_out":8247,"duration_ms":79102,"concrete_test":"Run the released code (https://github.com/Jam1ezhang/DYTO) on VideoMME 'w/o subs' using LLaVA-NeXT-Image-7B with N=100 and Z=3680, and on IntentQA using InternVL2-34B with the same settings. Compare the obtained accuracy to the values in Table 1 and Table 3 (VideoMME 7B: 41.2 vs 42.7; IntentQA 34B: 67.5 vs 67.3). If the code reproduces one value but not the other, that identifies a reporting error. If it reproduces neither, the published tables are not faithful to the implementation. Additionally, check whether Table 3's ablation rows are based on the same evaluation protocol; the authors should clarify and correct the discrepancies before the SOTA claim can be evaluated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that DYTO outperforms both fine-tuned and training-free methods rests entirely on the accuracy numbers in Tables 1-3. However, identical configurations report different results across those tables. In Table 1, DYTO-7B achieves 41.2 on VideoMME, but in Table 3 it achieves 42.7 for the same 7B LLaVA-NeXT-image model with the same dynamic configuration. Likewise, DYTO-34B scores 67.5 on IntentQA in Table 1 but 67.3 in Table 3. These are not different evaluation settings or ablations; they are the same rows repeated, and the numbers do not match. This internal inconsistency means the experimental record is unreliable: if a single reported score cannot be reproduced within the paper itself, then the head-to-head comparisons that support 'superior performance' and 'state-of-the-art' cannot be taken at face value. The reader's weakest assumption about the CLS-based clustering and the 'second-largest' rule is secondary: even if that rule were arbitrary, Table 3 shows DYTO 'w/o Clustering' still beats SlowFast on most benchmarks, so the claimed gains do not collapse solely on that assumption. But if the reported numbers are inconsistent, the entire empirical case is undermined. This is a load-bearing issue because no amount of methodological plausibility can substitute for internally consistent evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DYTO, a training-free framework for zero-shot video understanding built on image-based multimodal large language models. It first uniformly samples N=100 frames, uses per-frame [CLS] tokens with a temporally weighted distance (Eq. 1) to build a 1-NN graph (Eq. 2), recursively merges clusters, and selects the 'second-largest' clustering as the event segmentation. It then applies a bipartite token merging scheme (inspired by ToMe) to compress each cluster's visual tokens to a fixed budget Z, with a merge ratio r=Z/K. The method is evaluated on structured VQA (NExTQA, EgoSchema, IntentQA, VideoMME, MVBench) and open-ended VQA (MSVD-QA, MSRVTT-QA, TGIF-QA, ANet-QA, Video-ChatGPT) benchmarks across several base model families and sizes, with the central claim that DYTO achieves state-of-the-art zero-shot performance, even surpassing video-fine-tuned models.","tokens_in":15042,"tokens_out":3051,"duration_ms":28465,"significance":"If the empirical claims were reliable, DYTO would be a useful contribution: it is a training-free, conceptually simple method that adaptively selects frames and compresses tokens, with code available and fixed hyperparameters reported. The idea of using hierarchical clustering for keyframe selection and bipartite token merging is plausible and could inform future work on efficient video MLLM inference. However, the paper's headline claim of superiority over fine-tuned models is contradicted by its own tables, and the reported numbers are internally inconsistent across duplicate rows. These issues prevent the paper from making a trustworthy case for its main contribution.","major_comments":[{"comment":"The abstract and Section 4.3 claim that DYTO achieves 'superior performance compared to both fine-tuned and training-free methods' and 'outperforms all the training-free approaches as well as all the fine-tuned models across all benchmarks.' This claim is directly contradicted by Table 1: the fine-tuned LLaVA-OneVision-7B scores 79.4 on NExTQA, while DYTO-7B scores 65.7; similarly, in Table 2 the training-free SlowFast-34B scores 67.1 on MSRVTT-QA, while DYTO-34B scores 66.2. The headline statement is therefore false by the paper's own reported numbers, and the manuscript should either be substantially revised to a more modest claim or the experiments must be re-run and reported accurately.","section":"Abstract and Section 4.3, Table 1"},{"comment":"The experimental record is internally inconsistent. DYTO-7B is reported to score 41.2 on VideoMME in Table 1 but 42.7 in Table 3 for the same configuration (LLaVA-NeXT-image-7B, dynamic, Z=3680). Likewise, DYTO-34B scores 67.5 on IntentQA in Table 1 but 67.3 in Table 3. These are not different evaluation settings or ablations; they are duplicate rows with conflicting values. This inconsistency means the head-to-head comparisons supporting the state-of-the-art claim cannot be taken at face value, and the authors must reconcile the numbers or provide the raw evaluation logs to establish which values are correct.","section":"Tables 1 and 3"},{"comment":"The central design choice of selecting the 'second-largest' clustering result as the final segmentation is asserted without justification or sensitivity analysis. The ablation in Table 3 shows that removing the clustering module ('w/o Clustering') yields nearly identical performance to the full DYTO model, and even slightly better on NExTQA-7B (65.6 vs 65.7). This suggests that the claimed benefit of the hierarchical clustering is not clearly demonstrated, and the paper should provide an analysis of how the choice of clustering selection affects downstream accuracy, or temper the claim that dynamic keyframe selection is the source of the reported gains.","section":"Section 3.1, Eq. (2) and Table 3"},{"comment":"The comparison with fine-tuned models is not apples-to-apples and the paper does not disclose the computational cost of the fine-tuned baselines in the same setup. While the authors compare against methods with different frame counts and base models, the abstract's blanket statement that DYTO 'outperforms both fine-tuned and training-free methods' goes beyond what the table actually shows. Even where DYTO beats a fine-tuned model, the margins are often small and not accompanied by variance or significance testing, so the claim of a 'new state-of-the-art' is not statistically grounded.","section":"Section 4.3 and Tables 1-2"}],"minor_comments":[{"comment":"The title 'Scalabity on Base Model Size' contains a typo; it should be 'Scalability on Base Model Size'.","section":"Section 5.1 title"},{"comment":"The table header contains 'LLaVA-NeXT-iamge' instead of 'LLaVA-NeXT-image', and several entries are missing spaces between scores (e.g., '3.482.96' and '3.842.70').","section":"Table 2"},{"comment":"The text contains an incomplete word: 'verall, DyTo consistently outperforms' should be 'Overall, DYTO consistently outperforms'.","section":"Section 5.3"},{"comment":"The method name appears inconsistently as 'DYTO', 'DyTo', and 'D YTO' across the paper; please standardize.","section":"Throughout"},{"comment":"The text in the figure includes an artifact '1st-IN' and a partially merged word 'NIteration'; these should be cleaned up to make the pipeline diagram interpretable.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The internal inconsistency in Tables 1 and 3 is the most serious issue: if the authors cannot reproduce their own numbers, the entire empirical case collapses. I would ask the editor to require the authors to release the raw evaluation outputs or a clear explanation of how the two tables were generated. The claim of outperforming fine-tuned models is also overstated and should be corrected independently of the rerun."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on DYTO. The core recipe—hierarchical clustering on CLS tokens to pick keyframes, then ToMe-style bipartite merging to a fixed token budget—is a sensible, genuinely training-free combination that I don't think has been put together for video QA before. Credit where due: the hyperparameters are fixed and reported, there's no fitting to test benchmarks, and the evaluation spans ten benchmarks and several backbones. That's useful.\n\nBut the paper's central claim is false by its own tables. On NExTQA, fine-tuned LLaVA-OneVision-7B scores 79.4 while DYTO-7B scores 65.7. On MSRVTT-QA, SlowFast-34B scores 67.1 while DYTO-34B scores 66.2. So 'superior to both fine-tuned and training-free methods' does not hold. The gains over training-free baselines are real but modest—one to three points—and sometimes negative.\n\nThe bigger problem is internal inconsistency. The same DYTO-7B configuration reports 41.2 on VideoMME in Table 1 and 42.7 in Table 3; DYTO-34B reports 67.5 on IntentQA in Table 1 and 67.3 in Table 3. These are the same rows, not ablations. If the paper can't reproduce its own numbers in the same manuscript, the entire empirical case is suspect. That is not a minor typo; it's load-bearing because the SOTA claim rests entirely on those accuracy numbers.\n\nThe 'second-largest' clustering rule is also under-justified, and the ablation shows w/o Clustering sometimes beats full DYTO (NExTQA, both sizes). So the story is more nuanced than the abstract suggests. I'd also want error bars or at least multiple seeds, and a reconciliation of the conflicting numbers.\n\nWho is this for? People building training-free video QA pipelines will find the method worth looking at. But as a state-of-the-art claim, it doesn't hold up.\n\nRecommendation: send to peer review—the method is plausible and the scope is substantial—but require the authors to reconcile the inconsistent numbers and reframe the claims to match the evidence. A serious referee can get this into shape, but only if the data is fixed first.","headline":"A sensible training-free recipe, but the paper's own tables contradict its SOTA claim and contain conflicting numbers; needs major revision before it can be trusted.","tokens_in":15567,"tokens_out":2507,"would_cite":false,"duration_ms":23264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DYTO shows that a frozen image-based multimodal model, with dynamic frame clustering and bipartite token merging, can match or surpass video-fine-tuned models on zero-shot video question answering.","keywords":["zero-shot video understanding","multimodal large language models","token merging","hierarchical clustering","keyframe selection","visual token compression","training-free","video question answering"],"falsifier":"Take a video with two visually distinct but semantically one continuous event and compare DYTO's accuracy against a uniform-sampling baseline on a QA task about that event; if the clustering splits the video into two clusters and the token budget is halved per frame, accuracy on the continuous event should drop below the uniform baseline, which would show that the dynamic clustering does not always help. Alternatively, replace the second-largest rule by an oracle that tries all cluster counts on a benchmark such as VideoMME; if the oracle does not beat the fixed rule, the heuristic is not the driver of the gains.","tokens_in":14489,"feed_emoji":"🎬","tokens_out":6551,"duration_ms":56405,"temperature":0.7,"pith_summary":"DYTO is a training-free framework that performs zero-shot video question answering by dynamically selecting which frames to keep and how many visual tokens each selected frame contributes to the language model. The paper argues that fixed uniform sampling and pooled token averaging, used by prior training-free methods, lose both event-level and spatial detail; instead DYTO segments the sampled frames into event clusters using per-frame [CLS] tokens with temporal weighting, then compresses the visual tokens of each cluster to a fixed total budget through bipartite token merging. Evaluated on ten video QA benchmarks, DYTO reports accuracy gains over the strongest training-free baselines and, on several benchmarks, higher accuracy than models fine-tuned on video data. The claim matters because it implies that a frozen image-based MLLM, with clever input selection and token compression, can approach or exceed video-specific training, and that the deciding factor is how tokens are allocated, not additional supervision.","feed_headline":"No training needed: dynamic token merging tops zero-shot video QA","feed_subtitle":"Frozen image-based MLLMs beat video-fine-tuned models by choosing key frames and merging tokens adaptively.","key_machinery":"The machinery is a two-stage dynamic token allocation. Stage one computes a temporally weighted frame-distance matrix $W_t(i,j)=(1-\\langle v_i,v_j\\rangle)\\cdot|t_i-t_j|/N$ from L2-normalized [CLS] tokens, thresholds it into a 1-NN graph, takes connected components as candidate clusters, recursively merges clusters by feature-average similarity, and selects the second-largest clustering as the final segmentation, so the number of keyframes K adapts to the video's event structure. Stage two, inspired by token merging (ToMe), partitions each sampled frame's patch tokens into two sets, scores token pairs by head-averaged cosine similarity, and greedily pairs and pools the most similar tokens, leaving $R_0 - Z/K$ tokens per frame so that the total output to the language model is exactly $Z$ tokens. The dynamic per-cluster budget is the load-bearing device: videos with few distinct events keep more tokens per frame, while videos with many events trade per-frame detail for temporal coverage.","core_discovery":"The paper's central claim is that zero-shot video understanding improves when token allocation is dynamic rather than uniform. Using the [CLS] token of each frame and a temporally weighted distance that multiplies cosine dissimilarity by relative timestamp distance, DYTO builds a 1-NN graph whose connected components segment the video into K event clusters; it then samples representative frames from each cluster and runs a bipartite token-merging step that keeps $R_0 - Z/K$ tokens per frame, holding the total number of visual tokens at a constant $Z$. The method is completely training-free and model-agnostic, and on NExTQA, EgoSchema, IntentQA, VideoMME, and MVBench the paper reports that DYTO systematically beats the training-free baselines IG-VLM and SlowFast-LLaVA and at 34B scale also exceeds several video-fine-tuned models, with additional gains on open-ended MSVD-QA, MSRVTT-QA, TGIF-QA, ANet-QA, and Video-ChatGPT benchmarks.","pith_inferences":["The 'second-largest clustering' rule is a heuristic; DYTO does not derive it from the downstream task, and a cluster-count selection criterion could potentially improve segmentation further.","The token-budgeting principle generalizes: any video LLM could allocate its context window proportionally to event complexity, which may also help streaming or real-time settings with hard token limits.","Most open-ended gains are reported via GPT-assisted quality scores; human evaluation of temporal consistency would test whether the qualitative event-coverage advantage translates to perceived quality.","Because DYTO is a front-end only, it could in principle be stacked on top of video-fine-tuned models to improve their input selection as well, which the paper does not test."],"forward_implications":["Zero-shot video understanding can be improved through input selection and token compression alone, removing the need for video-level fine-tuning data and computation.","Sampling a larger number of frames (100 or more) and compressing them to a fixed token budget helps long-video tasks, because the clustering selects frames that cover distinct events instead of spreading tokens uniformly.","The accuracy gap between training-free and video-fine-tuned models narrows: at 34B scale DYTO outperforms several fine-tuned models on structured video QA benchmarks.","The method transfers across different image-based MLLM backbones and scales with model size, with larger backbones showing larger gains when combined with DYTO."],"supporting_citations":[{"why":"IG-VLM is the strongest training-free baseline that DYTO compares against; it assembles frames as an image grid for a VLM.","marker":"[11]"},{"why":"FreeVA is a training-free baseline and provides the GPT-3.5-Turbo evaluation protocol used for open-ended benchmarks.","marker":"[35]"},{"why":"SlowFast-LLaVA is the primary training-free baseline and the starting point of the ablation study, using short-long sampling with pooling.","marker":"[39]"},{"why":"ToMe supplies the token-merging algorithm that DYTO's bipartite merging step is inspired by.","marker":"[2]"},{"why":"Sarfraz et al. provides the first-neighbor relations used to build the 1-NN graph for the hierarchical clustering.","marker":"[31]"}],"fun_headline_variants":["Training-free DYTO: dynamic token merging wins zero-shot video","Merge, don't fine-tune: DYTO sets zero-shot video record","Zero-shot video: token merging beats fine-tuning without any training","DYTO: adaptive token merging boosts zero-shot video understanding","No fine-tuning, optimal tokens: DYTO leads zero-shot video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that a frame's [CLS] token, together with temporally weighted cosine distance and the choice of the second-largest connected-component clustering, reliably divides a video into the events that matter for the question being answered.","fun_headline_variants_meta":{"raw":{"variants":["Training-free DYTO: dynamic token merging wins zero-shot video","Merge, don't fine-tune: DYTO sets zero-shot video record","Zero-shot video: token merging beats fine-tuning without any training","DYTO: adaptive token merging boosts zero-shot video understanding","No fine-tuning, optimal tokens: DYTO leads zero-shot video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1580,"prompt_tokens":919,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":572}},"tokens_in":535,"tokens_out":661,"duration_ms":6662,"temperature":1.0,"reasoning_tokens":572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:13:03.945723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video with two visually distinct but semantically one continuous event and compare DYTO's accuracy against a uniform-sampling baseline on a QA task about that event; if the clustering splits the video into two clusters and the token budget is halved per frame, accuracy on the continuous event should drop below the uniform baseline, which would show that the dynamic clustering does not always help. Alternatively, replace the second-largest rule by an oracle that tries all cluster counts on a benchmark such as VideoMME; if the oracle does not beat the fixed rule, the heuristic is not the driver of the gains.","supporting_citations":[{"cited_title":"An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM, 2024","cited_arxiv_id":null,"evidence_quote":"IG-VLM is the strongest training-free baseline that DYTO compares against; it assembles frames as an image grid for a VLM."},{"cited_title":"Freeva: Offline mllm as training-free video assistant","cited_arxiv_id":null,"evidence_quote":"FreeVA is a training-free baseline and provides the GPT-3.5-Turbo evaluation protocol used for open-ended benchmarks."},{"cited_title":"SlowFast-LLaV A: A Strong Training-Free Baseline for Video Large Language Models, 2024","cited_arxiv_id":null,"evidence_quote":"SlowFast-LLaVA is the primary training-free baseline and the starting point of the ablation study, using short-long sampling with pooling."},{"cited_title":"Effi- cient parameter-free clustering using first neighbor relations","cited_arxiv_id":null,"evidence_quote":"Sarfraz et al. provides the first-neighbor relations used to build the 1-NN graph for the hierarchical clustering."}],"review_version":1}