{"id":"4750984f-34f1-4372-b02e-2baf24b4240c","arxiv_id":"2501.15513","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A small video model with a group resampler reduces video input to a few hundred tokens and beats several 7B models on common video benchmarks.","lead":"This paper presents TinyLLaVA-Video, a 3.6-billion-parameter video understanding model that compresses an entire video into a small set of learned visual tokens using a group resampler. It trains in one day on eight A100 GPUs and reports scores above several 7-billion-parameter models, making efficient video AI more accessible to researchers with limited compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 16-frame configuration (512 queries, 16 groups) is the image-level baseline, so the headline fixed-frame results do not clearly exercise the proposed group resampler.","rationale":"The reader's weakest assumption is benchmark selection: the group configuration was chosen by maximizing accuracy on the same four benchmarks used for the final comparison. That is a legitimate concern about overfitting and lack of held-out validation. However, an even more immediate issue is internal to the paper's own configuration reporting. Under Eq. 9 with 16 frames and 16 groups, the resampler is mathematically equivalent to the image-level baseline that the paper claims to supersede; the paper labels this as '(I)' in Fig. 12 and Table 9, and the best 16-frame row in Table 9 is that image-level setting. Consequently, the 16-frame headline numbers in Table 2, if they use this configuration, do not demonstrate the benefit of the video-level group resampler. The 1fps row does use a genuine group resampler and may still support the method, but the paper needs to state explicitly which variant produces each reported number and provide a controlled comparison where the true group resampler beats its image-level counterpart at the same total query budget. This does not overturn the reader's CONDITIONAL verdict, but it sharpens the condition: the authors must disambiguate the architecture behind the headline results. I found no reason to suspect dishonesty; the issue is ambiguity and a mismatch between the claimed mechanism and the configuration used for some reported results.","tokens_in":15214,"tokens_out":9219,"duration_ms":80842,"concrete_test":"Inspect the released training/eval configs for the 'TinyLLaVA-Video 3B 16' row in Table 2: if the number of groups equals the number of sampled frames (16), that variant is exactly the image-level method (Eqs. 1-3), so it cannot support the group-resampler claim. Then rerun the 16-frame, 512-query comparison with group counts 8, 16, and 32 on a held-out benchmark (not used for selection) and report whether any true video-level grouping outperforms the image-level setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proposed video-level group resampler in Eqs. 6-9 splits both the concatenated visual sequence and the learnable queries into M groups. With 16 uniformly sampled frames and M=16 groups, each group receives exactly one frame's tokens, making the operation identical to the image-level resampler of Eqs. 1-3. The paper itself marks this setting as equivalent to image-level in Fig. 12 ('512-16-Group=16(I)'). Yet the best 16-frame row in Table 9 is exactly this setting ('16 512 16 32', with Video-MME 55.9/46.6/38.4 and LVB 42.4), and Table 2's 'TinyLLaVA-Video 3B 16' row appears to use it. Meanwhile, the true video-level setting for 16 frames (group=8, each group covering 2 frames) scores worse (VM 54.4/44.4/38.7, LVB 36.4 in Table 9), which runs against the claim that grouping across frames improves temporal comprehension. The 1fps row does use group=16 (about 4 frames per group), but the paper does not clearly separate which headline comparisons rest on the group resampler versus the image-level fallback. The central attribution is therefore not settled: at the fixed-frame 512-query budget used in the main comparison, the proposed mechanism is not the one that wins.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TinyLLaVA-Video, a ~3.6B-parameter video LMM built from Qwen2.5-3B and SigLIP. Its proposed video-level group resampler (Eqs. 6-9) splits the concatenated visual token sequence and the learnable queries into M groups before cross-attention, with the aim of reducing visual token count, avoiding query redundancy, and improving temporal comprehension. The authors train on about 888K video-text samples in roughly one day on 8 A100-40G GPUs and report that the resulting model outperforms several 7B open-source video LMMs on Video-MME, MVBench, MLVU, and LongVideoBench (Table 2). The paper also presents controlled ablations (Figs. 5, 8, 10, 11, 12; Table 9) intended to isolate the effect of the resampler and to justify the choice of group count and number of queries.","tokens_in":15551,"tokens_out":11221,"duration_ms":86032,"significance":"If fully supported, the paper would offer a useful, resource-efficient baseline for small video LMMs, and the controlled comparison of three resampling schemes is a reasonable way to study token-reduction mechanisms. The open code and weights, the modest training budget, and the detailed per-benchmark tables are concrete strengths. However, the central attribution of the headline gains to the video-level group resampler is not currently established: the 16-frame, 512-query configuration used in the main comparison is equivalent to the image-level baseline, while the genuinely cross-frame configurations at that budget perform worse on several benchmarks. The significance of the work is therefore conditional on a re-analysis that separates image-level-equivalent settings from true cross-frame grouping and that addresses the test-set selection issue.","major_comments":[{"comment":"In the 16-frame, 512-total-query configuration used for the 'TinyLLaVA-Video 3B 16' row of Table 2, the setting 'Group=16, Query-per-group=32' in Table 9 is identical to the image-level resampler of Eqs. (1)-(3), because with 16 frames and 16 groups each group receives exactly one frame's visual tokens; the paper's own Fig. 12 marks this case as '512-16-Group=16(I)'. The genuinely cross-frame configuration at the same budget, Group=8 (two frames per group), scores lower on Video-MME (average 45.83 vs 46.97), LongVideoBench (36.4 vs 42.4), and MLVU (51.0 vs 52.5). The 1fps row does use cross-frame grouping, but the paper does not clearly separate which headline comparisons rest on the group resampler versus the image-level fallback. Consequently, the abstract's central claim that the video-level group resampler yields the reported gains is not supported by the 16-frame results in Table 2.","section":"Section 3.2, Table 9, Fig. 12"},{"comment":"The text states that 'the model achieves the best performance when each group of queries is responsible for learning information from approximately 2 to 4 frames, as shown in Fig. 12.' This generalization is not supported by Table 9: for 16 uniformly sampled frames at 512 total queries, the best average accuracy is obtained with Group=16 (one frame per group), while Group=8 (two frames per group) is worse by about two points on average; the 2-4 frame sweet spot holds only at the 128 and 256 total-query budgets. The paper should either restrict the claim to lower query budgets or explain why the effect reverses at the budget used in the final model.","section":"Section 3.2, Fig. 12, Table 9"},{"comment":"The group count M and total query count are selected by maximizing average accuracy on the same four benchmarks (Video-MME, MVBench, MLVU, LongVideoBench) that are later used for the headline comparison against 7B models in Table 2 and Tables 5-8. No held-out validation set, cross-validation, or multiple-seed evaluation is reported, and the Table 2 caption uses the word 'significantly' without any statistical test. This selection-on-test-set procedure makes it difficult to determine how much of the reported advantage over 7B models is a genuine property of the resampler rather than a result of tuning to these specific benchmarks. Please report a validation-based selection protocol or a separate held-out comparison.","section":"Section 3.2, Table 9, Figs. 11-12"}],"minor_comments":[{"comment":"The first sentence contains a grammatical error: 'Through Large Multimodal Models have achieved' should be 'Though Large Multimodal Models have achieved'.","section":"Abstract"},{"comment":"The sentence 'The code and weights is available' should read 'The code and weights are available'.","section":"Abstract"},{"comment":"The phrase 'PE and Concat represent Position Encoding and Concatenate, respectively' should be 'Positional Encoding and Concatenation' for grammatical consistency.","section":"Fig. 3 caption"},{"comment":"The table is difficult to read because the #Group and #Query-per-group columns are not clearly aligned with the rows in the rendered text; please format the table so that each row explicitly shows the group count and the number of queries per group.","section":"Table 9"},{"comment":"The phrase 'under the same settings' should state explicitly that the total number of queries is 512 and that the naive video-level resampler is used in those comparisons, so that readers can relate the architecture ablations to the main results.","section":"Section 4.2, Tables 10-12"},{"comment":"The 'zeroing out' experiment measures robustness to query removal; the authors should clarify why retaining 25% of queries at roughly 95% performance indicates redundancy rather than, for example, the LLM's ability to compensate for missing tokens.","section":"Section 3.2, Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the attribution of Table 2's 16-frame results to the group resampler. If the authors re-run the headline comparison with a configuration that genuinely groups across frames (e.g., the 1fps setting with group=16) and clearly report that the 16-frame row is the image-level fallback, the paper could be publishable after major revision. I would also recommend asking for a validation split or multiple seeds for the hyperparameter selection, given that the paper's own Table 9 shows the cross-frame advantage is not consistent across query budgets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. TinyLLaVA-Video does something real: a 3.6B model trainable in one day on 8 A100-40G GPUs, with code and weights released. The video-level group resampler—split the concatenated visual sequence and the learnable queries into M groups before cross-attention—is a simple, plausible mechanism, and the ablations in Fig. 10 and Table 9 give it genuine support at 128 and 256 total queries. The attention visualizations and the query-zeroing experiments are good evidence that naive global resampling spreads attention too thinly.\n\nThe problem is the headline configuration. With 16 uniformly sampled frames and M=16 groups, each group gets exactly one frame's tokens, which is the image-level resampler. The paper itself labels this 'Group=16(I)' in Fig. 12. Table 9 shows the best 16-frame, 512-query row is precisely that setting; the true video-level grouping (group=8, two frames per group) scores worse on Video-MME and LongVideoBench. So the Table 2 row that beats several 7B models at 16 frames is image-level in disguise. Only the 1fps row uses real grouping (about four frames per group), and there the gains over image-level are not isolated because no image-level 1fps baseline is shown.\n\nThe selection of group size and query count on exactly the four test benchmarks, without error bars or seeds, further blurs whether the margins over 7B models generalize. That said, the paper is not sloppy: the ablations are controlled, the data curation is documented, and the efficiency numbers are credible. The missing piece is a clean comparison of true group resampling vs. image-level at the same 512-query budget and same frame sampling, especially at 1fps. A minor issue: Qwen2-VL's learned resampler, which also handles video-level token groups, is not discussed or cited.\n\nWho's this for? Researchers building cheap video LMMs will get useful recipes and a reasonable resampler analysis. The paper deserves referee time—send it to peer review—but the central claim needs to be re-framed or re-supported. I'd ask for the missing ablation, error bars on the main table, and a clearer separation of which settings actually use the proposed method.","headline":"Useful empirical paper, but the headline 16-frame results use the image-level equivalent of the proposed resampler, so the central attribution is unsettled.","tokens_in":16044,"tokens_out":4387,"would_cite":false,"duration_ms":35819,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With a group resampler, a 3.6B video model beats several 7B models on four benchmarks while training in one day.","keywords":["video understanding","large multimodal models","group resampler","visual token compression","temporal reasoning","efficient training","lightweight models"],"falsifier":"Evaluate the chosen 512-query/16-group model against the image-level baseline on a video QA benchmark not used during development; if the 2–4-frame-per-group configuration no longer beats per-image resampling, the reported advantage is an artifact of benchmark selection.","tokens_in":15041,"feed_emoji":"🎬","tokens_out":10402,"duration_ms":79292,"temperature":0.7,"pith_summary":"This paper claims that a lightweight large multimodal model with about 3.6 billion parameters can outperform several open-source 7B video understanding models on Video-MME, MVBench, MLVU, and LongVideoBench, while training in roughly one day on eight A100-40G GPUs. The key mechanism is a video-level group resampler that splits the concatenated visual token sequence and the learnable queries into M groups before cross-attention, keeping the total number of visual tokens fixed and small regardless of frame count. Compared with per-frame resampling, this design lets each query attend only to a short temporal chunk of the video, which the paper argues avoids attention redundancy and preserves temporal order information. If the claim holds, strong video understanding does not require a 7B backbone or large-scale training data: the released 3.6B model with 888K training samples is competitive with larger models on standard benchmarks.","feed_headline":"A 3.6B video model beats 7B rivals with fewer tokens","feed_subtitle":"Grouped resampling shrinks visual tokens, captures temporal order, and trains in one day on eight A100s.","key_machinery":"The video-level group resampler is a cross-attention connector placed between a vision encoder and a small language model. It first concatenates the per-frame visual features into one long sequence and applies a linear projection. Then it splits both the projected visual sequence and the learnable query set into M equal groups, runs cross-attention group-wise with each query group attending only to its corresponding visual chunk, and concatenates the outputs. This keeps the total visual token count fixed and controllable (for instance 512) even when the number of sampled frames grows, prevents the attention redundancy seen in naive video-level resampling, and, because each group spans several consecutive frames, gives the model a way to register temporal dependencies that per-frame resampling misses.","core_discovery":"The central claim is that the video-level group resampler—which splits the concatenated visual token sequence and the learnable queries into M groups before cross-attention—significantly reduces the number of visual tokens while preserving the efficiency of query learning and enhances the understanding of temporal information (Section 3.2, Equations 6–9). The paper shows that with the chosen configuration (512 total queries, 16 groups, each group covering about 2–4 frames), the model achieves higher accuracy than both image-level resampling and a naive video-level resampler that lets every query attend to the whole sequence. It further reports that the resulting 3.6B model surpasses several open-source 7B-parameter video LMMs on Video-MME, MVBench, MLVU, and LongVideoBench, using a two-stage training pipeline with a pre-training stage on 397K video-caption pairs and a fine-tuning stage on 491K dialogue samples.","pith_inferences":["The group-splitting idea could generalize to compressing the visual tokens within a single frame, not just across frames, which may help high-resolution image inputs that currently rely on image-level resampling.","Because the group configuration was tuned on the four benchmark suites used for the headline comparison, the reported margins should be re-verified on a held-out benchmark before treating them as general.","The fixed token budget makes the design a natural fit for streaming inference, where the model could process video in temporal chunks and accumulate a constant number of tokens over time.","A cheaper alternative—replacing group-cross-attention with per-chunk pooling or local attention—might achieve similar temporal coverage; the paper does not test this, but the analysis suggests the grouping itself, not the attention mechanism, is the active ingredient."],"forward_implications":["A video model can process arbitrarily long videos with a fixed, small token budget instead of scaling tokens with frame count.","The same architecture, trained on roughly 900K samples, matches or exceeds several 7B open-source models on video QA benchmarks, so parameter count and data scale are not the only route to video understanding.","The group size is a tunable hyperparameter: the paper's ablation shows a sweet spot around 2–4 frames per group, with both too-few queries per group and image-level grouping hurting accuracy.","Training cost drops to about one day on eight A100-40G GPUs (25 hours for the group resampler configuration), making video LMM training feasible on modest clusters."],"supporting_citations":[{"why":"Introduced the idea of resampling visual features before an autoregressive language model; the paper treats it as the origin of connector-based token reduction.","marker":"[1]"},{"why":"A representative image-level resampler that compresses each frame into a few tokens; used as the main comparison baseline.","marker":"[16]"},{"why":"A recent LMM that also reduces video tokens via image-level resampling plus a pre-fusion module; used as a baseline on MVBench and MLVU.","marker":"[35]"},{"why":"Provides the attention-redundancy analysis that motivates the group-resampler design.","marker":"[4]"},{"why":"Supplies the two-stage training pipeline and the small-scale LMM framework that the paper builds on.","marker":"[38]"},{"why":"The Video-MME benchmark is one of the four evaluation suites; the paper's short/medium/long video numbers come from it.","marker":"[7]"},{"why":"The LongVideoBench evaluation is one of the four suites; it tests long-context video-language understanding.","marker":"[28]"},{"why":"The MVBench evaluation is one of the four suites; it provides the temporal-action-sequence cases used to show group resampling captures order.","marker":"[15]"},{"why":"The MLVU evaluation is the fourth suite; it tests long-video understanding with holistic and detail tasks.","marker":"[39]"}],"fun_headline_variants":["3.6B video model beats 7B rivals via group resampler","Group resampler shrinks tokens, boosts temporal video LMMs","TinyLLaVA-Video: 3.6B LMM trains in a day, outperforms 7B","Fewer tokens, better video understanding: group resampler wins","One-day training, 3.6B params, top results on video benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The group configuration (512 total queries, 16 groups) was chosen by maximizing average accuracy on the same four benchmarks later used for the headline comparison, and the paper assumes this selection does not materially overfit those test sets.","fun_headline_variants_meta":{"raw":{"variants":["3.6B video model beats 7B rivals via group resampler","Group resampler shrinks tokens, boosts temporal video LMMs","TinyLLaVA-Video: 3.6B LMM trains in a day, outperforms 7B","Fewer tokens, better video understanding: group resampler wins","One-day training, 3.6B params, top results on video benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001331,"raw_usage":{"total_tokens":5434,"prompt_tokens":987,"completion_tokens":4447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":4337}},"tokens_in":603,"tokens_out":4447,"duration_ms":28445,"temperature":1.0,"reasoning_tokens":4337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:12:11.734008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the chosen 512-query/16-group model against the image-level baseline on a video QA benchmark not used during development; if the 2–4-frame-per-group configuration no longer beats per-image resampling, the reported advantage is an artifact of benchmark selection.","supporting_citations":[{"cited_title":"Llama-vid: An image is worth 2 tokens in large language models","cited_arxiv_id":null,"evidence_quote":"A representative image-level resampler that compresses each frame into a few tokens; used as the main comparison baseline."},{"cited_title":"On attention redundancy: A comprehensive study","cited_arxiv_id":null,"evidence_quote":"Provides the attention-redundancy analysis that motivates the group-resampler design."}],"review_version":1}