{"id":"fa10cd5a-a98d-4ac5-99b2-f7e1b8704dcc","arxiv_id":"2412.05185","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A plug-and-play linear video tokenizer converts existing image-based LLMs into video-understanding LLMs by condensing frames into weighted-average tokens while preserving image capabilities.","lead":"LinVT is a plug-and-play module that turns image-only AI assistants into video assistants by condensing frames into a few tokens that are weighted averages of the original image tokens. Tested with six multimodal LLMs, it reports top scores on video QA and long-video benchmarks, though some zero-shot results reuse training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported zero-shot SOTA on four video QA benchmarks is invalid because those datasets appear in the Stage-2 training set (Table 1); the evaluation must be redone on held-out data.","rationale":"The reader's stated weakest assumption (linearity) is a conceptual concern about mechanism, but the more concrete and damaging issue is evaluation hygiene: several 'zero-shot' benchmarks are in the training set. The paper's abstract and Section 1 assert state-of-the-art video understanding, and the zero-shot comparisons in Table 7 are a central pillar of that claim. Since the training data are disclosed in Table 1, the contamination is directly verifiable from the manuscript itself, making it the most load-bearing concern. The reader's rationale already flags this issue, but their formal 'weakest_assumption' field focuses on linearity; hence partial agreement. The appropriate outcome remains conditional acceptance: the method may still be effective on genuinely held-out benchmarks, but the zero-shot claims must be corrected and the experiments redone. I recommend no change to the reader's verdict.","tokens_in":80,"tokens_out":2562,"duration_ms":36607,"concrete_test":"Retrain LinVT after removing from Stage 2 the four QA datasets ActivityNet-QA, MSVD-QA, MSRVTT-QA, and TGIF-QA (and NExT-QA if it is also claimed as zero-shot), then re-evaluate Table 7. If accuracy on those benchmarks drops materially or falls below published baselines, the claimed zero-shot SOTA does not hold; report the corrected numbers and re-assess the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of state-of-the-art zero-shot performance rests on benchmarks that are explicitly included in the training mixture. Table 1 lists Stage-2 instruction-tuning data containing ActivityNet-QA (29k), MSVD-QA (30k), MSRVTT-QA (109k), TGIF-QA (71k), and NExT-QA (34k). Yet Section 4.2 (Table 7) reports these same datasets as 'zero-shot' open-ended QA benchmarks: MVD-QA, MTT-QA, Act-QA, and TGIF-QA. The model has therefore seen the exact evaluation datasets during training, so these results are in-domain, not zero-shot. This invalidates the zero-shot label and materially weakens the headline SOTA claim, since several of the largest gains (e.g., LinVT-Qwen2-VL reaching 80.2% on MSVD-QA) are measured on training data. The only genuinely held-out evidence comes from benchmarks such as MVBench, Video-MME, and EgoSchema, which do not support the same strength of claim. Unless the training data are corrected or the evaluation rerun on truly unseen splits, the core contribution is not adequately validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LinVT, a plug-and-play video tokenizer module that converts image-level LLMs into video-LLMs after training on video data. The module is built on two principles: linearity (output tokens are convex combinations of input visual tokens) and representative information condensation (achieved via spatio-temporal significance scoring, top-k selection, multi-scale pooling, and text-conditioned aggregation). The authors integrate LinVT into six recent image-LLMs (Aquila, BLIP-3, InternVL2, Mipha, Molmo, Qwen2-VL) and report state-of-the-art results on a range of short, long, and open-ended video QA benchmarks, along with evidence that image understanding is preserved.","tokens_in":22991,"tokens_out":3319,"duration_ms":35286,"significance":"If the claims are validated, LinVT offers a lightweight, broadly compatible method for adapting image-LLMs to video without destroying their image-level competence. The ablations are systematic (SVR/TTA roles, multi-scale variants, text conditioning, design choices) and the compatibility across six diverse backbones is a strength. Independent evidence from benchmarks not present in the training mixture (MVBench, Video-MME, EgoSchema, MLVU, LongVideoBench) is reported, which partially supports the method's generality. However, the headline 'zero-shot' results in Table 7 and the associated SOTA claims are undermined by the fact that the four evaluation datasets also appear in the training set (Table 1), so the strength of the central contribution is not adequately validated as presented.","major_comments":[{"comment":"The four 'zero-shot' open-ended QA benchmarks in Table 7 (MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA) are all listed in the Stage-2 instruction-tuning dataset in Table 1. The model is therefore trained on those exact datasets before being evaluated on them, so the results are in-domain, not zero-shot. This invalidates the 'zero-shot' label and materially weakens the state-of-the-art claims in the abstract and introduction. The authors must either re-run the evaluation on truly held-out benchmarks or clearly disclose that the train splits of these benchmarks were used and remove the 'zero-shot' terminology; the same issue affects NExT-QA, which appears both in Table 1 and Table 8.","section":"Section 4.2, Table 7 vs Table 1"},{"comment":"The paper's central design principle, that linear (convex) combination of the image-LLM's visual tokens preserves visual-language alignment, is not directly tested. The full LinVT encoder includes nonlinear operations (top-k token selection and attention scoring), so the overall map is not linear. The ablation in Table 3 compares the original intermediate layer against a randomly initialized one, which changes many factors beyond linearity; it does not isolate linearity as the cause of preserved image ability. Please provide a more controlled comparison, e.g., against a nonlinear tokenizer of similar capacity, or soften the claim to state that the specific linear-aggregation design preserves the empirical value distribution and is conjectured to help alignment.","section":"Section 3.2, Eq. (1) and Table 3"},{"comment":"The description of Stage-1 training is underspecified: 'Training is conduct under contrastive loss and cross-entropy loss' is vague. Which contrastive loss is used, over what pairs, and how is the cross-entropy loss applied? This level of detail is necessary for reproducibility, especially since the stage is claimed to be essential for the final performance.","section":"Section 3.3, Training recipe"}],"minor_comments":[{"comment":"Typo: 'at rish' should be 'at risk' in the paragraph on excessive visual tokens.","section":"Section 1"},{"comment":"Heading 'Sate-of-The-Art' should be 'State-of-the-Art'.","section":"Section 4.2"},{"comment":"The caption 'Importantce' is a typo for 'Importance'.","section":"Tables 3 and 10"},{"comment":"The notation '80.2 / 4.4' is used throughout but is not explained in any caption; please define that the first number is accuracy and the second is a score (e.g., CIDEr or consistency) in the table notes.","section":"Table 7 and other tables"},{"comment":"Claiming 267M parameters is 'negligible compared to the original billion-size image-LLMs' is misleading for the 1.6B Mipha model, where it is a substantial fraction; please report the relative overhead per backbone.","section":"Section 6 (supplementary)"},{"comment":"The description of spatio-temporal significance scoring is incomplete: 'applying self-attention within the corresponding visual token' does not explain how scalar significance scores are produced from the attention maps; please specify the exact computation.","section":"Section 3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The zero-shot terminology is common in parts of the video-LLM literature, but this manuscript is unusual in that the exact benchmark names appear verbatim in the training-data table. Even if the authors used only the train splits of those benchmarks for training and the test splits for evaluation—which the paper does not state—most readers and reviewers would not regard that as 'zero-shot'. This is a load-bearing framing issue that can be fixed by reframing the results as in-domain evaluation and highlighting the held-out benchmarks as the primary evidence, or by re-running on unseen benchmarks. The underlying method appears plausible and the ablations are thorough, so I do not recommend rejection, but the manuscript cannot be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know is that LinVT is a genuinely useful plug-and-play video tokenizer, and the paper does the compatibility work properly. It takes six different image-LLMs, adds the module, trains only on video data, and shows consistent gains on held-out benchmarks like MVBench, Video-MME, EgoSchema, and LongVideoBench. The ablation study is systematic: SVR, TTA, multi-scale variants, text conditioning, and the alignment-preservation claim are each tested with a randomized-projector baseline. That last ablation is the most convincing part. Keeping the original intermediate layer preserves image QA performance, while randomizing it destroys it.\n\nThe soft spot is evaluation hygiene. Table 7 is titled zero-shot open-ended QA, but the datasets in that table — ActivityNet-QA, MSVD-QA, MSRVTT-QA, TGIF-QA — are in the Stage-2 instruction-tuning mix in Table 1. So those are in-domain numbers, not zero-shot. The SOTA claim on those benchmarks is undermined. The genuinely held-out results still show solid gains, but they are smaller and more varied. The paper needs to either retrain without those datasets, evaluate on unseen splits, or honestly relabel the results and re-argue the case.\n\nThere is also a smaller inconsistency in the core claim. The paper says the output is \"strictly the weighted average of (part of) the input visual tokens,\" but the top-k selection is discontinuous and the attention weights come from a softmax over QK^T, which is nonlinear in the input tokens. So the overall map is not actually linear. The design intuition is reasonable — outputs live in the convex hull of selected tokens and plausibly preserve alignment — but the \"strict linear\" phrasing overstates it.\n\nSo the method itself looks solid and the ablations are thorough. The paper is worth serious referee time, but it is not ready to accept as is. The zero-shot claims need to be fixed, and the linearity language softened. After that it would be a decent paper for a video-LLM venue. I would cite it for the compatibility study and the alignment-preservation ablation, but not for the zero-shot numbers.\n\nRecommendation: send it to peer review, with an editor's note asking the authors to address the training/eval overlap and the linearity wording.","headline":"Useful plug-and-play video tokenizer, but the headline zero-shot numbers are contaminated because the test benchmarks appear in the training set.","tokens_in":23471,"tokens_out":1754,"would_cite":true,"duration_ms":18479,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A linear video tokenizer turns image LLMs into video LLMs","keywords":["video understanding","large language models","video tokenizer","linear transformation","token compression","vision-language alignment","multi-scale pooling","video instruction tuning"],"falsifier":"Run the paper's alignment-preservation ablation, original versus randomly initialized intermediate layer, on a second image-LLM such as Molmo or Qwen2-VL with the same video-only recipe; if image-benchmark scores drop substantially even with the original projector kept, the linearity premise is falsified for that host. A sharper test: because top-k selection makes the map non-linear, replace the softmax scoring with a fixed linear weighting and observe whether image preservation and video accuracy both change.","tokens_in":22488,"feed_emoji":"🎥","tokens_out":6606,"duration_ms":57017,"temperature":0.7,"pith_summary":"The paper introduces LinVT, a plug-and-play module that bolts onto an existing image-based large language model and, after training on video data only, gives that model video understanding without erasing its image skills. The central bet is that if the video tokenizer's outputs are weighted averages of the image model's own visual tokens, the original image–language alignment survives training, so the model can be taught video on video data alone. The authors test the module on six different image-LLMs, including some never trained on video, and report state-of-the-art scores across short, long, and open-ended video QA benchmarks, while image benchmarks stay at baseline levels. If the bet holds, any mature image-LLM can be upgraded to video with a small, compatible adapter instead of retraining a video model from scratch.","feed_headline":"A linear video tokenizer turns image LLMs into video LLMs","feed_subtitle":"LinVT trains only on video data, keeps image skills intact, and tops benchmarks across six base models.","key_machinery":"The central object is the linear aggregation map $\\Phi_{\\mathrm{Lin}}(T_v, Q_{v,t}) = \\mathrm{softmax}(W_q Q_{v,t} (W_k T_v)^\\top) \\cdot T_v$, where $T_v$ are the multi-scale visual tokens and $Q_{v,t}$ are text-conditioned scale-specific queries. Because the value matrix is the identity, with no residual connection and no value projection, every output token is a convex combination of input visual tokens, which is the linearity that is supposed to protect the image-LLM's existing visual–language alignment during video-only training. The complementary mechanism is representative information condensation: spatio-temporal significance scoring plus hard top-$k$ selection reduces thousands of frame tokens to a manageable set, and shifted-window average pooling creates multiple temporal scales so both short events and long dependencies survive. Together the two mechanisms, linearity for knowledge preservation and condensation for tractability, are what let a frozen image-LLM take on video tasks.","core_discovery":"LinVT claims that a frozen image-LLM can become a strong video-LLM through a tokenizer whose outputs are strictly convex combinations, weighted averages, of part of the frame-level visual tokens the image model already produces. The module has two parts: a Spatio-Temporal Visual Token Refiner (SVR), which scores tokens by spatio-temporal significance, keeps the top-k, and pools them at multiple temporal scales, and a Text-conditioned Token Aggregation (TTA), which uses scale-specific learnable queries that attend to visual tokens and to the user's text, then aggregates via a linear layer that computes softmax attention maps directly over the input values with no residual connections. Because each output is a weighted sum of input tokens, the argument goes, the original vision–language alignment is preserved; because only the informative tokens survive top-k selection, the video is condensed to a fixed, LLM-friendly length. Trained in two stages on video data only, alignment then instruction tuning, LinVT-based models, with LinVT-Qwen2-VL in particular, reach top accuracy on MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA, MVBench, VideoMME and long-video benchmarks, while the image-benchmark ablation shows performance roughly equal to the untouched image-LLM.","pith_inferences":["Because the paper tests the alignment-preservation premise only on Mipha-1.6B, a natural extension is to run the same original-versus-randomized-projector ablation on the other five base models; if image scores drop on any of them, the linearity premise is host-specific rather than general.","The full tokenizer is not literally linear in all input tokens because hard top-k selection and softmax scoring are discrete and nonlinear steps; replacing the softmax scoring with a fixed linear weighting would test whether the preservation effect comes from the convex-combination form or simply from freezing the image backbone.","The state-of-the-art comparisons are against open-source video-LLMs, so a direct comparison with commercial closed models such as GPT-4o or Gemini on the same video benchmarks would clarify how close video-only-trained LinVT gets to much larger systems.","The recipe suggests a broader principle: any modality adapter whose outputs are convex combinations of a frozen backbone's tokens may inherit that backbone's alignment, a hypothesis that could be tested on other modality transfers such as audio or point clouds."],"forward_implications":["Any mature image-LLM can be upgraded to video understanding by inserting LinVT and training only on video data, without a from-scratch video pretraining pipeline.","Because the base LLM, vision encoder, and intermediate layers stay frozen during alignment, with only LoRA on the LLM during instruction tuning, the upgrade cost is small relative to training a video-LLM from scratch.","Video-only training with LinVT preserves image-level ability, so a single model can serve both image and video tasks after the upgrade.","The multi-scale and text-conditioned design lets the same module handle events of different durations and question-relevant content, which is why long-video benchmarks improve along with short-video QA.","LinVT also boosts models that already have some video capability, such as InternVL2, Aquila, and Qwen2-VL, showing it is an add-on rather than merely an adapter for image-only models."],"supporting_citations":[{"why":"Aquila (Infinity-MM) is one of the six image-LLMs LinVT is plugged into, a video-compatible model used in the ablation and comparisons.","marker":"[19]"},{"why":"BLIP-3 (xGen-MM) is one of the six image-LLMs; an image-only model that LinVT turns into a video-LLM.","marker":"[75]"},{"why":"InternVL2 is one of the six image-LLMs; a video-compatible model where LinVT improves video benchmarks.","marker":"[8]"},{"why":"Mipha is the small image-LLM used for the alignment-preservation ablation and the design ablations.","marker":"[86]"},{"why":"Molmo is one of the six image-LLMs; the basis of the LinVT-Molmo-7B flagship comparison in Figure 1.","marker":"[12]"},{"why":"Qwen2-VL is the base of the best-performing LinVT model in the paper's experiments.","marker":"[65]"},{"why":"Q-Former/InstructBLIP is the contrast architecture for TTA, which differs by outputting linear combinations and using multi-scale text-conditioned queries.","marker":"[11]"},{"why":"Perceiver in Flamingo is the contrast architecture for TTA, where Perceiver-style cross-attention is adapted with scale-specific queries and no residual.","marker":"[1]"}],"fun_headline_variants":["LinVT plugs into any image LLM and adds video vision","One linear layer gives any image LLM video understanding","Linear tokenizer LinVT: image LLMs watch videos too","LinVT: video tokens from image tokens, no model retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that outputting weighted averages of the image model's own visual tokens will preserve its image–language alignment through video-only training, so image ability is never traded away for video ability, a premise the paper supports with a single model (Mipha-1.6B) rather than a proof or a multi-model ablation.","fun_headline_variants_meta":{"raw":{"variants":["LinVT plugs into any image LLM and adds video vision","One linear layer gives any image LLM video understanding","Linear tokenizer LinVT: image LLMs watch videos too","LinVT: video tokens from image tokens, no model retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000864,"raw_usage":{"total_tokens":3765,"prompt_tokens":981,"completion_tokens":2784,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2714}},"tokens_in":597,"tokens_out":2784,"duration_ms":20816,"temperature":1.0,"reasoning_tokens":2714,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:57.629497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's alignment-preservation ablation, original versus randomly initialized intermediate layer, on a second image-LLM such as Molmo or Qwen2-VL with the same video-only recipe; if image-benchmark scores drop substantially even with the original projector kept, the linearity premise is falsified for that host. A sharper test: because top-k selection makes the map non-linear, replace the softmax scoring with a fixed linear weighting and observe whether image preservation and video accuracy both change.","supporting_citations":[],"review_version":1}