{"id":"f60d2ef4-45d6-43f5-861a-bf3a1b0a3177","arxiv_id":"2411.11066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A training-free video LLM that fuses a low-resolution thumbnail grid with uniformly sampled visual tokens achieves state-of-the-art results among zero-shot video-language models.","lead":"This paper proposes TS-LLaVA, a way to let an image-only AI model answer questions about videos without any video-specific training. It combines a small low-resolution thumbnail of a few frames with a random sampling of visual tokens from all frames, and beats prior zero-shot video models on several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SOTA among training-free video LLMs depends on treating PLLaVA as training-based, but the cited PLLaVA paper is titled 'Parameter-free' and is usually categorized as training-free; in the paper's own MVBench table, PLLaVA-34B scores 58.1 vs TS-LLaVA-34B's 52.6.","rationale":"The paper is otherwise carefully controlled, with useful ablations and an honest appendix on GPT-evaluation unreliability. My concern is not that the method is weak internally, but that the headline claim is defined by an unargued exclusion. The reader focused on cross-paper benchmark comparability; that is a real issue, but the more specific and decisive weakness is the PLLaVA categorization. If PLLaVA is training-free, then the paper's own Table 5(a) contradicts the abstract. If PLLaVA is genuinely training-based, the concern disappears, and the remaining issue is the reader's comparability caveat. A single verification step (reading the PLLaVA paper/repo or re-running it) settles the matter, so conditional acceptance with mandatory verification is appropriate rather than outright rejection.","tokens_in":22098,"tokens_out":16775,"duration_ms":164918,"concrete_test":"Check arXiv:2404.16994 and its released repository for any video-training step (train scripts, updated checkpoints, video instruction data). If PLLaVA is training-free, re-run PLLaVA-34B and TS-LLaVA-34B on MVBench with identical prompts, frame sampling, and answer parsing, and recompute average accuracy. If PLLaVA's 58.1 is reproduced, the abstract's SOTA claim among training-free video LLMs is not supportable and the paper should reclassify PLLaVA and revise the claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is 'new state-of-the-art among training-free video LLMs' (Abstract). This depends on the training-free/trained boundary drawn in Table 5(a). PLLaVA [44] is placed above the dashed line and Sec. 4.2 asserts it 'is further trained on video data.' However, the cited paper is titled 'PLLaVA: Parameter-free LLaVA Extension from Images to Videos' and the method is widely treated as training-free. If that is correct, the claimed SOTA fails on the paper's own numbers: TS-LLaVA-34B MVBench Avg is 52.6 while PLLaVA-34B is 58.1, and TS-LLaVA-7B (45.5) trails PLLaVA-7B (46.6). The manuscript provides no evidence of a training stage in PLLaVA, no re-run of PLLaVA under its protocol, and no discussion of why the cited title should be read as 'trained on video data.' The abstract's headline comparison to GPT-4V is also less informative if a training-free baseline already beats it by a larger margin. This is a factual/classification issue, not merely a difference in benchmark provenance; if the authors cannot substantiate the 'further trained' claim, the SOTA statement must be restricted or retracted.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TS-LLaVA, a training-free video LLM that compresses visual tokens by combining a thumbnail image (a grid of a small number of equidistant frames) with uniformly sampled visual tokens from all frames. The authors first compare five token-compression strategies on Video-MME, then introduce Thumbnail-and-Sampling and evaluate it on Multiple Choice VideoQA benchmarks (NExT-QA, EgoSchema, IntentQA), MVBench, and MLVU, also reporting Open-Ended VideoQA and text-generation results in an appendix. The central claims are that TS-LLaVA achieves new state-of-the-art among training-free video LLMs, that TS-LLaVA-34B outperforms GPT-4V on MVBench, and that it is comparable to Video-LLaMA2-72B on MLVU.","tokens_in":22391,"tokens_out":11095,"duration_ms":80394,"significance":"If the SOTA claim is substantiated, the paper is a strong empirical contribution: it offers a simple, token-efficient compression schedule that works with frozen image LLMs, with well-matched ablations at fixed token budgets. The authors provide code, and their repeated GPT-evaluation case study in Sec. 7.2 is a useful methodological caution about open-ended video QA scoring. Even if the SOTA framing needs revision, the controlled comparisons among compression strategies and the positive MVBench/MLVU results are informative for the training-free video LLM literature.","major_comments":[{"comment":"The paper classifies PLLaVA as a training-based video LLM (\"is further trained on video data,\" Sec. 4.2) and places it above the training-free dashed line in Table 5, but the cited reference [44] is titled \"PLLaVA: Parameter-free LLaVA Extension from Images to Videos\" and presents a pooling-based method without a video-training stage. If PLLaVA is training-free, then on the paper's own Table 5a, TS-LLaVA-7B (45.5) and TS-LLaVA-34B (52.6) are both below PLLaVA-7B (46.6) and PLLaVA-34B (58.1), respectively, and the abstract's \"new state-of-the-art among training-free video LLMs\" claim is false. The authors must either provide evidence of a video training stage in PLLaVA or reclassify it and revise the SOTA statements accordingly.","section":"Sec. 4.2, Table 5"},{"comment":"The headline comparisons to PLLaVA, GPT-4V, and Video-LLaMA2-72B rely on accuracy numbers taken from prior papers (footnote in Table 5) rather than on re-runs under the authors' own frame-sampling, prompt, and decoding protocol. Because the SOTA claim depends on these cross-paper numbers, the authors should either re-run the closest competitor (PLLaVA) under their protocol or explicitly restrict the SOTA claim to methods evaluated in a controlled setting; otherwise the comparison is not apples-to-apples.","section":"Sec. 4.2, Tables 5 and 7"}],"minor_comments":[{"comment":"The Grid strategy uses 576 visual tokens while the other strategies use 2304 tokens; the text notes this, but the comparison would be cleaner if the token-budget mismatch were presented as a separate analysis rather than within the same matched-budget table.","section":"Sec. 3.1, Table 1"},{"comment":"Many design choices (50 input frames, 6 frames per thumbnail, one thumbnail, 3456 visual tokens) appear to be selected using the same benchmarks on which the final results are reported; the paper should disclose whether any held-out validation was used, and report variance or multiple seeds where feasible.","section":"Sec. 4.3 and Appendix Sec. 7.3"},{"comment":"The notation RM−V is slightly ambiguous because M is the total visual-token budget and V is the number of thumbnail tokens; defining M_s = M−V as the sampled-token budget would make the equation easier to follow.","section":"Eq. (1) and Fig. 3"},{"comment":"There are minor spelling inconsistencies (e.g., \"TS-LLaV A\" in the header vs. \"TS-LLaVA\" in the text; \"Video-LLAMA2\" vs. \"Video-LLaMA2\"), and the appendix Table 8 refers to \"colored text\" that is not visible in the grayscale version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the classification of PLLaVA as a training-based method. The cited paper's title and abstract describe a parameter-free extension, so the onus is on the authors to substantiate their claim. If they cannot, the abstract and Sec. 4.2 should be revised to remove or qualify the SOTA claim. The paper's internal ablations are sound and the method appears useful, so the issue is fixable within revision scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper for its ablation study, not for its headline. The matched-budget comparison of five compression strategies (Concat, Pooling, Grid, Grids, Sampling) on Video-MME is the most useful part. The finding that uniform sampling beats pooling, and that adding a thumbnail image on top of sampling helps, is a genuinely useful engineering insight. The token efficiency comparison against SF-LLaVA (better accuracy with 2304 tokens vs 3680) is also well done, and the appendix's demonstration that GPT-assisted evaluation is noisy—and the authors' decision to de-emphasize open-ended results—shows honesty. Code is available.\n\nThe soft spots are real. The paper classifies PLLaVA as a training-based video LLM and claims TS-LLaVA is SOTA among training-free methods. But the cited PLLaVA paper is titled 'Parameter-free LLaVA Extension' and is generally treated as training-free. The authors give no evidence for 'further trained on video data.' On the paper's own MVBench numbers, PLLaVA-34B scores 58.1 vs TS-LLaVA-34B's 52.6, so if PLLaVA is training-free, the SOTA claim fails. That's a load-bearing classification error, not a provenance nitpick. The abstract should be amended or defended.\n\nOther concerns are minor. The GPT-4V and Video-LLaMA2 comparisons rest on numbers lifted from other papers, with no re-run under identical prompts; that's standard for the field but worth a caveat. The hyperparameters were chosen on the same benchmarks that are then reported, so some overfitting risk exists. No variance or multiple seeds are reported, but the differences between strategies are large enough that this is unlikely to change the ranking. The MLVU 'comparable to 72B Video-LLaMA2' phrasing is generous: the 34B model trails the 72B model on average accuracy (43.0 vs 45.6), though it wins on several sub-tasks.\n\nWho is this for? Anyone building zero-shot video LLMs or working on visual token compression. The comparison study is a useful reference point. It deserves a serious referee: the methodology is sound, the code is out, and the main flaw is correctable. I'd suggest the authors re-run PLLaVA under their protocol or cite the PLLaVA paper's own claims about being parameter-free, and then restrict the SOTA statement accordingly. After that revision, this would be a solid acceptance.","headline":"A strong, well-controlled empirical baseline whose 'SOTA among training-free video LLMs' claim is undermined by the paper's own MVBench table if PLLaVA is training-free, as its title suggests.","tokens_in":22966,"tokens_out":6843,"would_cite":true,"duration_ms":56055,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a training-free video LLM can match video-trained giants by feeding a frozen image LLM a low-resolution thumbnail grid of a few frames plus uniformly sampled tokens from all frames, establishing a new state of the…","keywords":["training-free video LLM","visual token compression","thumbnail-and-sampling","video question answering","MVBench","MLVU","token efficiency"],"falsifier":"Run TS-LLaVA-34B and GPT-4V on the same MVBench instances with identical prompt templates and decoding settings, and TS-LLaVA-34B against Video-LLaMA2-72B on MLVU with identical settings; if the reported average gaps (52.6 vs 43.5 on MVBench and 43.0 vs 45.6 on MLVU) shrink, invert, or change qualitatively, the comparative claim fails while the internal ablations still stand.","tokens_in":21880,"feed_emoji":"🎬","tokens_out":5780,"duration_ms":51529,"temperature":0.7,"pith_summary":"The paper asks whether a powerful image-only LLM can understand video without any video-specific training, and answers yes if the visual tokens are organized correctly. It compares five token-compression strategies and finds that spatial pooling, the most common choice, is not very effective, while grid thumbnails and uniform token sampling each capture complementary strengths. The proposed TS-LLaVA combines both: a grid-view thumbnail made from a few equidistant frames gives a global summary, and uniformly sampled tokens from all frames preserve detail. On multiple-choice video QA benchmarks this training-free recipe sets a new state of the art, with the 34B version outperforming GPT-4V on MVBench and reaching accuracy comparable to Video-LLaMA2-72B on MLVU. The significance is that video understanding may not require large curated video-text datasets if token compression is designed well.","feed_headline":"Frozen image LLM beats GPT-4V on video QA with a thumbnail trick","feed_subtitle":"A training-free 34B video LLM built on a frozen image model rivals a 72B video-trained model on long-video QA.","key_machinery":"Thumbnail-and-Sampling token construction. Given N frames, select N_T equidistant frames (N_T ≪ N, even) to build one grid-view thumbnail image I_T, encoded to F_T ∈ R^V; all N frames are encoded and uniformly sampled to F_S ∈ $R^{{M−V}}$; the LLM receives [F_S, F_T]. The thumbnail supplies a low-resolution global summary that shines on spatial and temporal layout reasoning, while uniform sampling preserves high-resolution details from every frame that support action reasoning; combining them at the same total compression rate outperforms either stream alone.","core_discovery":"TS-LLaVA establishes that a frozen image LLM (LLaVA-v1.6, 7B or 34B) can be turned into a strong video QA system with no fine-tuning by replacing the visual token stream with a Thumbnail-and-Sampling schedule. From N uniformly sampled frames, a few equidistant frames are arranged into a single low-resolution grid image (the Thumbnail), whose 576 tokens summarize the video; the remaining token budget comes from uniformly sampling tokens across all N frames at full resolution. The two token groups are concatenated and passed to the LLM. The paper reports that this hybrid outperforms Concat, Pooling, Grid, Grids, and Sampling alone on Video-MME, and on NExT-QA, EgoSchema, IntentQA, MVBench, and MLVU it exceeds prior training-free methods, with the 34B model beating GPT-4V on MVBench average accuracy and matching Video-LLaMA2-72B on MLVU.","pith_inferences":["The headline comparisons to GPT-4V and Video-LLaMA2-72B are taken from published numbers, not matched re-runs; identical-prompt runs could shift the gaps, so a direct head-to-head is the natural next check.","The success of uniform sampling over pooling suggests spatial averaging throws away information that frozen LLMs need for temporal and action reasoning; this may extend to other long-context or multi-image settings.","Choosing thumbnail frames by content (e.g., keyframes) rather than equidistant spacing is a straightforward extension the paper does not explore.","Because the method is training-free and backbone-agnostic, it can be transplanted to newer image LLMs; transferring the schedule is a cheap test of whether the compression principle generalizes."],"forward_implications":["A frozen, image-only LLM can do competitive long-form video question answering without any video-text fine-tuning.","Token organization, not raw token count, drives training-free video performance: TS-LLaVA beats SF-LLaVA with roughly 60% of its visual tokens (2304 vs 3680).","The hybrid schedule inherits Grid's spatial and temporal reasoning and Sampling's action reasoning, improving both over either strategy on Video-MME.","The 34B model's MVBench average (52.6) and MLVU average (43.0) show that a smaller training-free model can stand comparison with much larger video-trained systems (GPT-4V and Video-LLaMA2-72B) on multiple-choice QA."],"supporting_citations":[{"why":"Grid-view image method that supplies the thumbnail idea and the NExT-QA evaluation protocol TS-LLaVA extends.","marker":"[13]"},{"why":"Previous state-of-the-art training-free video LLM whose slow-fast design and reported numbers are the main baseline to beat.","marker":"[45]"},{"why":"Video-MME benchmark used for the controlled strategy comparison that motivates the Thumbnail-and-Sampling design.","marker":"[10]"},{"why":"MVBench multitask benchmark, including per-task GPT-4V and PLLaVA numbers used in the comparison table.","marker":"[17]"},{"why":"MLVU long-video benchmark whose multi-choice results include the Video-LLaMA2-72B comparison target.","marker":"[53]"},{"why":"Video-LLaMA2-72B, the training-based video LLM whose MLVU accuracy TS-LLaVA-34B is claimed to match.","marker":"[5]"},{"why":"PLLaVA training-based method sharing the same backbone whose MVBench results are used for comparison.","marker":"[44]"},{"why":"LLaVA-v1.6, the frozen image LLM backbone used in all TS-LLaVA configurations.","marker":"[23]"},{"why":"GPT-4V system card, the proprietary model whose MVBench scores are the comparison supporting the claim of outperformance.","marker":"[30]"}],"fun_headline_variants":["No-fine-tune video LLM: thumbnail plus sampling tops GPT-4V","Thumbnail-sampling tops GPT-4V on video QA without any training","Frozen image LLM + thumbnail-sampling beats GPT-4V on MVBench","TS-LLaVA: training-free video QA that rivals 72B models","Thumbnail-and-sampling video LLM: zero fine-tuning, top scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparisons to GPT-4V and Video-LLaMA2-72B rest on benchmark numbers reported by other papers or official repositories rather than re-runs with identical prompts, frame sampling, and decoding settings; if those numbers are not directly comparable, the claimed superiority is not established.","fun_headline_variants_meta":{"raw":{"variants":["No-fine-tune video LLM: thumbnail plus sampling tops GPT-4V","Thumbnail-sampling tops GPT-4V on video QA without any training","Frozen image LLM + thumbnail-sampling beats GPT-4V on MVBench","TS-LLaVA: training-free video QA that rivals 72B models","Thumbnail-and-sampling video LLM: zero fine-tuning, top scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00099,"raw_usage":{"total_tokens":4239,"prompt_tokens":1032,"completion_tokens":3207,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":3109}},"tokens_in":648,"tokens_out":3207,"duration_ms":24049,"temperature":1.0,"reasoning_tokens":3109,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:56:19.099491+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TS-LLaVA-34B and GPT-4V on the same MVBench instances with identical prompt templates and decoding settings, and TS-LLaVA-34B against Video-LLaMA2-72B on MLVU with identical settings; if the reported average gaps (52.6 vs 43.5 on MVBench and 43.0 vs 45.6 on MLVU) shrink, invert, or change qualitatively, the comparative claim fails while the internal ablations still stand.","supporting_citations":[{"cited_title":"Mvbench: A comprehensive multi- modal video understanding benchmark","cited_arxiv_id":null,"evidence_quote":"MVBench multitask benchmark, including per-task GPT-4V and PLLaVA numbers used in the comparison table."},{"cited_title":"Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024","cited_arxiv_id":null,"evidence_quote":"LLaVA-v1.6, the frozen image LLM backbone used in all TS-LLaVA configurations."}],"review_version":1}