{"id":"710be132-de22-42c1-829c-1836d73cecf8","arxiv_id":"2504.14432","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":13,"one_line_summary":"ResNetVLLM claims state-of-the-art zero-shot video QA by training a non-pretrained ResNet jointly with a LLaVA-based LLM on instruction data, without releasing code or statistical details.","lead":"This paper proposes ResNetVLLM, a video question answering model that pairs a randomly initialized ResNet visual encoder with a large language model and claims state-of-the-art zero-shot performance on four video QA benchmarks. The results would matter if true, but the paper provides no code, no error bars, and several methodological gaps that leave the headline numbers unverifiable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stated training recipe is not executable as written: full LLaVA-transformer joint training for 50 epochs on 100k samples cannot fit in 2 V100s in 4 hours, and the warm-up phase has no defined loss.","rationale":"The reader's weakest_assumption is that the described two-stage training actually produces a transferable video encoder, and I agree this is the load-bearing premise. However, the most concrete reason to doubt it is not just the missing loss/projection details, but that the stated compute budget and optimizer recipe are internally inconsistent: full fine-tuning of a 7B LLaVA transformer on 100k samples for 50 epochs cannot be done on 2 V100s in 4 hours. This computational impossibility makes the reported results irreproducible regardless of the warm-up objective. The warm-up phase lacking any specified loss reinforces the same conclusion. I do not see a route to accept the SOTA claim without a corrected, executable training description and preferably released artifacts. The reader's verdict of REJECT therefore stands, though my emphasis shifts to the compute-budget inconsistency rather than the unspecified loss alone.","tokens_in":10412,"tokens_out":7143,"duration_ms":63522,"concrete_test":"Use standard FLOP counting and memory estimation to compute the theoretical minimum time and memory for the described Joint-Training: 50 epochs over 100k video samples with a 7B-parameter LLaVA transformer, ResNet encoder, AdamW optimizer, and 2 NVIDIA V100 GPUs. If the lower-bound runtime exceeds 4 hours by an order of magnitude, or if the model cannot fit in 32GB without offloading/LoRA, the reported training protocol is not reproducible as stated. Also verify whether the warm-up phase can be run at all without a specified loss or output head.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the two-stage training in Section IV-A to actually train the model, and that is the least secure condition. The paper states that in Joint-Training, the entire model, including the ResNet encoder and the LLaVA transformer, is trained for 50 epochs over the Video-ChatGPT-100K dataset (100k video-instruction pairs) with AdamW on 2 NVIDIA V100 GPUs in about 4 hours. LLaVA's transformer is a Vicuna LLM of 7B or 13B parameters; full AdamW fine-tuning of such a model requires far more than the 32GB HBM available on 2 V100s (model weights, gradients, and optimizer states alone exceed that capacity without offloading or LoRA). Even under optimistic FLOP counts, processing 5M sample-epochs with a 7B transformer would take not hours but days on 2 V100s. Separately, the Warm-Up phase says the ResNet is trained alone for 150 epochs with SGD, but never specifies a loss, labels, or an output head, so it has no well-defined optimization objective. If the training procedure cannot be executed as described, the SOTA results in Tables I and II cannot be attributed to the claimed method, and the central empirical claim is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ResNetVLLM, a video understanding framework that combines a randomly initialized 2D ResNet visual encoder with a LLaVA-based language model. It claims state-of-the-art zero-shot video question-answering results on MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, ActivityNet-QA, and the VideoChatGPT generative benchmark. Training is described as a two-stage procedure in which the ResNet is first warmed up alone and then the full model is jointly trained on Video-ChatGPT-100K for 50 epochs on two NVIDIA V100 GPUs in about four hours.","tokens_in":10700,"tokens_out":5104,"duration_ms":44174,"significance":"The reported findings, if reproducible, would be significant: a non-pretrained ResNet trained jointly with an LLM in a few hours outperforming models that rely on CLIP or pretrained video encoders would challenge common assumptions about pretraining. The paper uses standard benchmarks and compares against recent baselines, which is an appropriate experimental framing. However, the manuscript provides no code, checkpoints, evaluation scripts, error bars, or significance tests, and the training protocol as stated is both underdetermined and computationally implausible. Until those issues are resolved, the claimed state-of-the-art results are not verifiable and the significance of the contribution cannot be assessed.","major_comments":[{"comment":"The Warm-Up phase is specified only as training the ResNet for 150 epochs with SGD, a learning rate of 0.01, and a weight decay of 0.0001; no loss function, labels, output head, or data split is given. Without a defined objective, this phase cannot be executed or reproduced, and any resulting encoder is not well-defined. This is load-bearing because the central claim of learning from a non-pretrained encoder depends on this phase actually training the encoder.","section":"IV-A, Warm-Up phase"},{"comment":"The paper states that the entire model, including the ResNet encoder and the LLaVA transformer, is trained for 50 epochs over the 100K-pair Video-ChatGPT-100K dataset with AdamW on 2 NVIDIA V100 GPUs in approximately 4 hours. Under standard dense AdamW fine-tuning, the weights, gradients, and optimizer states of a 7B or 13B transformer exceed the memory of two V100s, and 5 million sample-epochs of such a model require far more compute than two V100s can deliver in four hours. The description therefore omits essential implementation details such as parameter-efficient adaptation (LoRA/adapters), offloading, gradient checkpointing, or reduced sequence length; as written, the experiment is not executable, and the results in Tables I and II cannot be attributed to the stated training protocol.","section":"IV-A, Joint-Training phase"},{"comment":"The visual features are described as flattened global ResNet features that are \"projected\" and merged with the tokenized text, but no projection layer, feature dimension, or injection mechanism into the LLaVA transformer is defined. This leaves the architecture underdetermined and makes it impossible to reproduce the method from the text alone.","section":"III-C, Model Formulation"},{"comment":"Training uses the Video-ChatGPT-100K instruction dataset, which is built from videos including ActivityNet, while evaluation includes ActivityNet-QA and the ActivityNet-200-based VideoChatGPT generative benchmark. The paper neither reports whether the training and evaluation videos overlap nor enforces disjoint splits for the zero-shot evaluation; if overlap exists, the zero-shot claim is compromised. The authors should state the overlap and provide a contamination analysis.","section":"IV-A and IV-C, ActivityNet overlap"},{"comment":"All reported numbers are single-run results without error bars, multiple seeds, or significance tests, and no code, checkpoints, or evaluation scripts are provided. Given the large claimed gaps over prior work (e.g., Table I Mean 3.55 vs. 2.42 for Video-ChatGPT, and Table II MSVD-QA 78.3 vs. 67.5 for BT-Adapter), this lack of reproducibility evidence is a central obstacle to accepting the empirical claims.","section":"Tables I and II"}],"minor_comments":[{"comment":"The phrase \"with a Large Language Model (LLM.\" is missing the closing parenthesis and should read \"(LLM).\".","section":"Abstract"},{"comment":"The term \"LLaV A\" appears with an unintended space; the standard spelling \"LLaVA\" should be used throughout.","section":"Sections III and IV"},{"comment":"The equations for CI, DO, CU, TU, and C define symbols such as fCI, wc, ws, Ci, and Si only in prose and are not algorithmically specified; please either provide precise computational definitions or defer to the official VideoChatGPT evaluation implementation.","section":"IV-B, Evaluation Metrics"},{"comment":"Bayesian Optimization is mentioned twice, but neither the objective function nor the hyperparameter search space is described, which makes the tuning procedure unreproducible.","section":"IV-A, Implementation Details"},{"comment":"The caption contains a duplicated article: \"THE THE ZERO-SHOT QUESTION-ANSWER EVALUATION\".","section":"Table II caption"}],"recommendation":"reject","confidential_remarks":"In my assessment, the manuscript's central empirical claims are not verifiable because the training procedure as written is both underdetermined and computationally implausible, and no code or artifacts are provided. I do not see a way to repair this within a normal revision short of a substantially rewritten experimental section with complete implementation details, corrected resource reporting, and data/code release. Therefore I recommend rejection; if the authors can provide those materials and reproducible results, a fresh submission would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2504.14432. It claims state-of-the-art zero-shot video QA with a randomly initialized ResNet feeding a LLaVA-style LLM, trained on two V100s in four hours. If those numbers were real, it would be a big deal. They are not supported by the paper as written, and the training recipe is not executable as described.\n\nThe idea itself is a fair variation on Video-ChatGPT: replace the pretrained CLIP encoder with an untrained ResNet, train end-to-end, and see what happens. The paper is readable, cites the relevant 2023 work, and uses standard benchmarks, reporting both generative and QA numbers. That is the extent of the credit I can give.\n\nThe soft spots are load-bearing. First, the Warm-Up phase says the ResNet is trained alone for 150 epochs with SGD, but never gives the loss, labels, or output head. That stage has no defined objective. Second, the projection from ResNet features into the LLaVA transformer is not specified beyond 'flattened global features.' Third, the compute claim is hard to believe: full fine-tuning of a Vicuna-scale LLM (even 7B) for 50 epochs on 100k video-instruction pairs needs far more than 32GB of V100 memory and far more than four hours on two GPUs. The stress-test arithmetic is right; it is off by days, not minutes. Fourth, the results are single-run, with no code, checkpoints, or evaluation scripts. Fifth, the ActivityNet-QA evaluation is likely contaminated because Video-ChatGPT-100K draws videos from ActivityNet. Finally, there is no ablation with a pretrained ResNet, so the 'non-pretrained encoder' is not established as the cause of anything.\n\nNothing here is a minor typo. The central empirical claim is unverified, and the method is underdetermined to the point that a reader could not reproduce the training run. The paper would need code, a full specification of both stages, a realistically sized fine-tuning recipe, and a clean evaluation protocol before it is worth referee time.\n\nWho gets value from it? Someone curious about the design space of video LLMs might note it as a promising direction, but it cannot be cited as evidence. I would not bring it to reading group, and I would desk reject it now.","headline":"The SOTA claim rests on a training recipe that cannot run as described, and the method section omits the loss and projection details, so the reported numbers are not supported.","tokens_in":11253,"tokens_out":4617,"would_cite":false,"duration_ms":38582,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A non-pretrained ResNet plus an LLM claims state-of-the-art zero-shot video understanding.","keywords":["zero-shot video understanding","video question answering","ResNet visual encoder","large language model","multi-modal LLM","Video-ChatGPT benchmark","cross-modal learning","video captioning"],"falsifier":"An independent reproduction using only the details given in the paper—100 frames per video at 6-frame intervals, random $224\\times224$ crops, a randomly initialized ResNet warm-up for $150$ epochs with SGD at learning rate $0.01$, joint training for $50$ epochs with AdamW at learning rate $0.00015$, and evaluation on MSVD-QA—would disprove the claim if the resulting accuracy does not approach the reported $78.3$.","tokens_in":10160,"feed_emoji":"🎬","tokens_out":9889,"duration_ms":80200,"temperature":0.7,"pith_summary":"ResNetVLLM is a proposed video-understanding model that joins a randomly initialized, non-pretrained ResNet visual encoder to a large language model, training them together on video-text pairs and then evaluating without seeing question-answer training data. The paper's central claim is that this design beats every compared video LLM on four open-ended video QA benchmarks—MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA—and on all five generative quality axes of the Video-ChatGPT benchmark. If true, expensive pretrained video encoders are unnecessary for strong zero-shot video understanding, and a simple two-stage recipe running about four hours on two GPUs can learn visual and semantic representations in one architecture. The paper introduces no new dataset; it trains on Video-ChatGPT-100K.","feed_headline":"A non-pretrained ResNet plus LLM tops video question-answering","feed_subtitle":"Paper reports state-of-the-art scores on four benchmarks after just four hours of training on two GPUs.","key_machinery":"The central object is the ResNetVLLM architecture: a vanilla, non-pretrained 2D ResNet visual encoder connected to a LLaVA-based LLM transformer, with [CLS] tokens and flattened global average-pooled features projected into the sequence the language model reads. The random weight initialization of the ResNet is the mechanism intended to guarantee zero-shot fairness by preventing prior video knowledge from entering the model. Training proceeds in two stages: a $150$-epoch warm-up that optimizes the ResNet alone with SGD, followed by a $50$-epoch joint stage that optimizes the ResNet and the LLM transformer together with AdamW.","core_discovery":"The paper's discovery claim is that a vanilla, randomly initialized 2D ResNet can serve as the visual encoder of a video LLM and outperform models built on pretrained video features in zero-shot settings. The model samples frames at regular intervals, extracts features with the ResNet, flattens the global average-pooled features, projects them into a LLaVA-based language model, and is trained first with a ResNet-only warm-up and then jointly with the LLM. The authors report state-of-the-art accuracy on MSVD-QA, MSRVTT-QA, TGIF-QA FrameQA, and ActivityNet-QA, and a 3.55 mean score on the Video-ChatGPT generative benchmark, above the previous best 2.42. They attribute the gains to joint learning of visual and semantic representations in a shared space, which they say minimizes information loss and makes the embeddings discriminative and semantically consistent.","pith_inferences":["Editorial inference: the magnitude of the reported gains suggests the LLM's language prior, trained on vast text, may carry more of the semantic and temporal load than the visual encoder, since the random ResNet is trained on only 100K video-text pairs for a few hours.","Editorial inference: because the model feeds flattened global ResNet features rather than tokenized spatial features, it likely captures global scene content better than fine-grained object details; a testable extension would probe the model on questions about small objects or spatial relations.","Editorial inference: the same random-encoder-plus-LLM design could be transferred to audio or sensor streams, although the paper only demonstrates video."],"forward_implications":["Pretrained video encoders become optional rather than necessary for competitive zero-shot video QA, lowering the barrier to building new video LLMs.","The two-stage recipe of a random-encoder warm-up followed by joint LLM fine-tuning becomes a reusable training template for other vision-language tasks.","The reported scores give the video-LLM community new reference points: $78.3$ on MSVD-QA, $63.5$ on MSRVTT-QA, $59.9$ on TGIF-QA, and $54.8$ on ActivityNet-QA.","A direct corollary is that frozen CLIP-style visual features are not required for strong video understanding, contrary to the usual design assumption."],"supporting_citations":[{"why":"supplies the Video-ChatGPT-100K training set, the benchmark's five generative quality axes, and the Video-ChatGPT baseline.","marker":"[14]"},{"why":"provides the LLaVA base whose transformer and visual-instruction tuning are reused as the language component.","marker":"[9]"},{"why":"defines the ResNet architecture used as the randomly initialized visual encoder.","marker":"[43]"},{"why":"provides the MSRVTT-QA and MSVD-QA datasets used for zero-shot QA evaluation.","marker":"[23]"},{"why":"provides the TGIF-QA FrameQA dataset used for zero-shot QA evaluation.","marker":"[24]"},{"why":"provides the ActivityNet-QA dataset used for zero-shot QA evaluation.","marker":"[25]"},{"why":"supplies the ActivityNet-200 videos underlying the generative performance benchmark.","marker":"[45]"},{"why":"FrozenBiLM is the prior zero-shot baseline against which the QA improvements are measured.","marker":"[47]"}],"fun_headline_variants":["From-scratch ResNet powers zero-shot video QA record","Video LLM with vanilla ResNet beats pretrained features","Non-pretrained ResNet + LLM sets new video QA bar","Zero-shot video QA: random ResNet outperforms pretrained"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the central claim to hold, the described two-stage training must be a well-defined procedure—with a concrete warm-up objective and a compatible way of feeding ResNet features into the language model—so that someone else could reproduce the reported scores.","fun_headline_variants_meta":{"raw":{"variants":["From-scratch ResNet powers zero-shot video QA record","Video LLM with vanilla ResNet beats pretrained features","Non-pretrained ResNet + LLM sets new video QA bar","Zero-shot video QA: random ResNet outperforms pretrained"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1415,"prompt_tokens":876,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":467}},"tokens_in":492,"tokens_out":539,"duration_ms":4984,"temperature":1.0,"reasoning_tokens":467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:48:39.009779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent reproduction using only the details given in the paper—100 frames per video at 6-frame intervals, random $224\\times224$ crops, a randomly initialized ResNet warm-up for $150$ epochs with SGD at learning rate $0.01$, joint training for $50$ epochs with AdamW at learning rate $0.00015$, and evaluation on MSVD-QA—would disprove the claim if the resulting accuracy does not approach the reported $78.3$.","supporting_citations":[{"cited_title":"Tgif-qa: Toward spatio- temporal reasoning in visual question answering,","cited_arxiv_id":null,"evidence_quote":"provides the TGIF-QA FrameQA dataset used for zero-shot QA evaluation."},{"cited_title":"Activi- tynet: A large-scale video benchmark for human activity understanding,","cited_arxiv_id":null,"evidence_quote":"supplies the ActivityNet-200 videos underlying the generative performance benchmark."},{"cited_title":"Zero-shot video question answering via frozen bidirectional language models,","cited_arxiv_id":null,"evidence_quote":"FrozenBiLM is the prior zero-shot baseline against which the QA improvements are measured."}],"review_version":1}