{"id":"256c3753-6285-49fd-986b-a19972a8d04b","arxiv_id":"2412.17415","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"For video question answering, feeding each frame together with question-aware captions of distant frames into an image-language model, then max-pooling per-frame answers, gives competitive open-model accuracy.","lead":"VidCtx answers video questions by combining what a vision-language model sees in one frame with a text description of a distant frame. It does this without any training or fine-tuning, and it improves accuracy on three video QA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The external SOTA claim rests on unre-run baseline numbers; the internal same-model ablations are sound, so the concern is real but bounded.","rationale":"The reader identified exactly the right soft spot: every point in Table I except VidCtx's row is from another paper, under settings that are not matched to VidCtx. This is a genuine threat to the 'state-of-the-art among open approaches' phrasing, because the base LMM is a major confound: LLaVA-1.6-Mistral-7B is a stronger image-language model than the captioners used in several earlier pipelines, and 64 frames is more than some baselines used. I therefore agree with the reader's weakest_assumption. At the same time, a stress-test should note what is not fragile: the paper's own controlled comparisons in Tables II and III used the same model and the same captions for both arms, so the contribution of adding visual frames and distant question-aware captions is internally consistent. Table IV's monotonic scaling also supports the frame-count claim. My only deviation from the reader's framing is emphasis: I would not let the unre-run baselines cast doubt on the mechanism, only on the precise magnitude of the SOTA margins. The conditional verdict remains appropriate; no verdict change is needed.","tokens_in":9296,"tokens_out":8352,"duration_ms":81843,"concrete_test":"Take the official Q-ViD code and adapt only the inference settings to match VidCtx: use LLaVA-1.6-Mistral-7B (4-bit) to generate question-aware captions from the same 64-frame sampling, feed the 64 captions to a text-only Mistral-7B with Q-ViD's answer prompt, and score first-token probabilities. Run this controlled Q-ViD variant on NExT-QA validation and IntentQA test, and compare against VidCtx under the same frames and captions. If the margin stays near +3% on both, the SOTA claim is substantiated; if it shrinks to ≤1%, the reported advantage is mostly base-model/FPS effects rather than the visual+distant-caption mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing uncertainty is in the comparative claim of Table I/Section IV-C: VidCtx is compared with Q-ViD, SeViLA, LLoVi, etc. using numbers copied from their original papers, which were produced with different backbone LMMs, frame counts, prompts, and aggregation. VidCtx uses 4-bit LLaVA-1.6-Mistral-7B and 64 frames on NExT-QA/IntentQA, while Q-ViD's published 12B result uses its own pipeline (older LLaVA captioner plus separate LLM reasoner) and likely different sampling. Thus the headline +4.4/+3.5 margins could be partly an artifact of the newer base model rather than the proposed context mechanism. This matters because the strongest_claim asserts state-of-the-art among open models. The same concern does not, however, threaten the internal mechanism: Table III compares Captions Only (67.3) vs VidCtx (70.3) at 32 frames with 'the same set of captions and same model', and Table II compares No Context (67.9) vs Distant Q-Aware Caption (70.7); both isolate the proposed additions. So the central qualitative claim is supported; the exact SOTA margins are not fully controlled. Minor additional caveat: aggregation choices (L1 norm vs softmax, max vs mean) and 64-frame sampling were selected on NExT-QA validation, so the absolute numbers may be slightly optimistic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VidCtx, a training-free video question-answering framework that combines visual evidence from individual frames with question-aware textual captions of distant frames. A pre-trained image LMM (LLaVA-1.6-Mistral-7B, 4-bit) is used both to generate captions for sampled frames and to produce frame-level multiple-choice decisions conditioned on the frame, the question, and a distant-frame caption. The frame-level decisions are aggregated by max pooling of L1-normalized first-token scores. The method is evaluated zero-shot on NExT-QA, IntentQA, and STAR, where it reports strong performance among open-model approaches. The paper contributes a simple, modular architecture, careful ablations of context choice and aggregation, and a qualitative analysis.","tokens_in":9596,"tokens_out":4488,"duration_ms":43420,"significance":"If the reported results hold, VidCtx is a valuable contribution to training-free VideoQA: it demonstrates that combining per-frame visual input with textual context from distant frames is complementary, and it does so with a cheap, reproducible pipeline that scales linearly in the number of frames. The internal ablations (Tables II, III, IV, V) are well designed and support the key qualitative claims (distant question-aware captions help; max pooling with L1 normalization helps). The release of code is a further strength. The main reservation is that the headline state-of-the-art comparison against Q-ViD, SeViLA, and other open approaches in Table I relies on numbers taken from their original papers under different backbones, frame counts, and prompts, so the exact margins are not fully controlled. The central mechanism, however, is supported by the same-model ablations.","major_comments":[{"comment":"The claim that VidCtx achieves state-of-the-art performance among open approaches on NExT-QA and IntentQA is based on accuracy numbers copied from the original papers of Q-ViD, SeViLA, LLoVi, and others. Those methods use different backbone LMMs (e.g., Q-ViD is 12B with a separate captioner and reasoner), different frame sampling strategies, and different prompt templates. Since VidCtx uses 4-bit LLaVA-1.6-Mistral-7B with 64 frames on NExT-QA/IntentQA and 32 on STAR, the reported +4.4% and +3.5% margins over Q-ViD may be partly attributable to the stronger base model rather than to the proposed context mechanism. To support the SOTA claim, the authors should re-run at least the most competitive open baselines under VidCtx's exact settings (same LMM, same frame count, same prompt), or explicitly qualify the comparison as non-controlled and use wording such as 'competitive' rather than 'state-of-the-art'.","section":"Section IV-C, Table I"},{"comment":"Equation (4) is the core aggregation mechanism, but its notation is ambiguous and potentially incorrect. The text in Section III-D says 'we consider the probability of just the first token,' yet Eq. (4) defines p(d_i = t) as a 'log probability score.' If p is a log probability, it is typically negative, and the L1 normalization in the denominator sums absolute values, producing normalized scores that can be negative. This is an unusual and unjustified operation. If instead p denotes a probability, the denominator should sum probabilities, not their absolute values. The authors should clarify exactly which quantity is used, how the first token is extracted (e.g., after prompting the model to output a single letter), and why the absolute-value L1 normalization is appropriate. This is essential for reproducibility of the central max-pooling mechanism.","section":"Section III-D, Eq. (4)"},{"comment":"All ablation results are reported as single-run top-1 accuracies without variance or significance testing. Several differences that drive design choices are small (e.g., 70.7 vs. 70.3 for distant vs. current caption in Table II; 70.7 vs. 70.6 for L1+max vs. softmax+max in Table V). If decoding is stochastic (e.g., temperature sampling), these differences could be within run-to-run noise. The authors should either report multiple runs with different seeds, provide evidence that inference is deterministic, or explicitly state that these differences are qualitative rather than statistically established.","section":"Section IV-D, Tables II-V"}],"minor_comments":[{"comment":"Equation (2) writes d_i = φ(v_i, c_{r(i)}, I_vqa), but I_vqa includes the question Q; this dependency is clear in the text but would be clearer if the equation also showed Q explicitly, for instance d_i = φ(v_i, c_{r(i)}, Q, I_vqa).","section":"Section III-C, Eq. (2)"},{"comment":"The special token F ('No Answer') is excluded from the argmax in Eq. (4), but the paper does not specify what happens if the LMM outputs F for every frame. A fallback policy (e.g., choose the highest-scoring option despite F, or predict a default) should be stated.","section":"Section III-D"},{"comment":"The sentence 'We generate 200 tokens per caption' should specify whether 200 is the maximum generation length or an exact target length; otherwise the captioning cost is not precisely described.","section":"Section IV-A"},{"comment":"The statement that related works such as LLoVi and Q-ViD 'scale quadratically' due to attention in the LLM is a simplification; the quadratic scaling is in the number of tokens, and those methods may truncate or chunk captions. A more precise description of the complexity comparison would be helpful.","section":"Section IV-C, Computational Complexity"},{"comment":"The aggregation method labeled 'Voting' is not defined in the text. Does it mean majority vote over the argmax of each frame's answer, or something else? Please define it.","section":"Section IV-D, Table V"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within scope for ICME and the internal ablations suggest the core idea is sound. The main gating issue for acceptance is the uncontrolled external comparison in Table I; if the authors can add at least a small-scale controlled re-run of Q-ViD or a similarly strong open baseline with the same LLaVA-1.6-Mistral-7B backbone and frame sampling, the SOTA claim would become credible. Otherwise, the claim should be softened. The Eq. (4) ambiguity must also be fixed for reproducibility. I see no citation or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: VidCtx is a clean, training-free VideoQA pipeline that pairs each frame's visual input with a question-aware caption from a distant frame, then max-pools L1-normalized first-token probabilities across frames. The internal evidence is solid: Tables II and III show the distant-caption mechanism and the visual-plus-caption combination are each worth real points over the same model without them. If you work on zero-shot video QA with image-only LMMs, this is worth a look.\n\nWhat's genuinely new is the specific combination: per-frame decisions with distant-frame captions as context, plus the normalization/pooling trick. It's not a big conceptual jump over Q-ViD, SeViLA, or Vamos, but the ablations isolate each component better than most papers in this space. The code is public, the method runs on one RTX 4090, and the frame-count scaling in Table IV is a nice practical result.\n\nThe main soft spot is the comparative claim in Table I. The +4.4/+3.5 margins over Q-ViD come from numbers taken from the original papers, not from re-running those baselines under VidCtx's exact conditions -- different backbone (LLaVA-1.6-Mistral-7B 4-bit), different frame counts, and a different captioner. Since Q-ViD uses a 12B model and its own captioning pipeline, part of the margin is likely just the newer base model. The paper doesn't hide this, but the wording \"outperforming Q-ViD\" goes beyond what is actually controlled. Another, smaller caveat: the frame count, distant offset, and aggregation choices are picked on the validation set, so the absolute numbers may be slightly optimistic.\n\nI don't think this is fatal. The internal ablations support the qualitative claim: the proposed context mechanism works, and the visual information adds value over captions alone. The uncertainty is about the size of the SOTA margin, not about the existence of an effect.\n\nThis paper is for people building practical zero-shot VideoQA systems with off-the-shelf LMMs, and for anyone comparing training-free vs trained video pipelines. It deserves a serious referee; the right outcome is likely acceptance after a controlled baseline re-run or a softened comparative claim.","headline":"VidCtx is a sound, training-free VideoQA pipeline whose internal ablations are convincing; the reported SOTA margin is plausible but not fully controlled because the baselines were not re-run.","tokens_in":10114,"tokens_out":2786,"would_cite":true,"duration_ms":26980,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VidCtx claims a training-free recipe that pairs each video frame with a question-aware caption of a distant frame and max-pools the per-frame answers, reaching the best open-model results on NExT-QA and IntentQA.","keywords":["large multimodal models","video question answering","zero-shot learning","question-aware captions","frame-level voting","max pooling","temporal reasoning","multiple-choice video QA"],"falsifier":"Re-run Q-ViD's caption-concatenation pipeline with the same LLaVa-1.6-Mistral-7B backbone, 4-bit quantization, 64-frame sampling, the same question-aware captions, and the same answer prompt; if its NExT-QA accuracy reaches or exceeds 70.7%, the paper's claimed +4.4% advantage over Q-ViD would disappear.","tokens_in":9107,"feed_emoji":"🎬","tokens_out":5641,"duration_ms":47282,"temperature":0.7,"pith_summary":"This paper argues that a single off-the-shelf image-language model can answer multiple-choice video questions well without any video training, if each sampled frame is paired with a question-aware caption of a distant frame and the per-frame choices are combined by max pooling. The claim is that visual and textual modalities are complementary, and that a caption placed half a video away gives the model enough temporal context to reason about causal and temporal questions. On NExT-QA and IntentQA this recipe reaches 70.7% and 67.1% accuracy, the best among open-model methods in the paper's comparison, and it stays competitive on STAR. The practical interest is that the approach is training-free, uses one 7B LMM for both captioning and answering, and scales linearly with the number of frames rather than quadratically.","feed_headline":"Frame-by-frame voting lifts zero-shot video QA by 4.4 points","feed_subtitle":"Pairs frames with distant captions, max-pools decisions, beats prior open models with no training.","key_machinery":"The load-bearing mechanism is the distant-frame pairing: for each sampled frame $i$ of $N$, the answer prompt receives the question-aware caption of frame $r(i) \\equiv i + \\frac{N}{2} \\pmod{N}$, together with an 'earlier' or 'later' temporal specifier. Captions are produced by prompting the same LMM with a short description instruction that includes the question. A special 'No Answer' token lets the model abstain on irrelevant frames, and the final decision is $y = \\arg\\max_{t \\in T-\\{F\\}} \\max_{0 \\le i < N} \\frac{p(d_i=t)}{\\sum_{k\\in T} p(d_i=k)}$, i.e. L1-normalized per-frame first-token probabilities passed through max pooling. This mechanism converts the temporal structure of the video into local pairwise constraints, and it is what lets the method scale to an arbitrary number of frames.","core_discovery":"VidCtx's central claim is that zero-shot multiple-choice video question answering improves when the answer about a frame is conditioned on the question-aware caption of a second frame located roughly half a video away, and when the resulting per-frame answer distributions are aggregated by L1-normalized max pooling. The paper reports 70.7% top-1 on NExT-QA, which is +4.4% over Q-ViD and the best among the open-model methods in its table; 67.1% on IntentQA, +3.5% over Q-ViD; and 51.1% average on STAR, second to VideoChat2, a model pre-trained on large video corpora. VidCtx also reports beating the GPT-3.5-based LLoVi pipeline while using a 7B open model. The authors take this as evidence that combining the visual signal of one frame with text from a distant frame captures temporal structure better than either captions-only or single-frame visual processing alone.","pith_inferences":["An untested but direct extension is to choose the paired frame adaptively, for example by caption similarity to the question or by the model's abstention signal, instead of always half a video away; the paper's own context ablations suggest the optimum may vary with video length.","The +3.5 to +4.4 point margins over Q-ViD rely on published baseline numbers; re-running those baselines under VidCtx's exact sampling, 4-bit quantization, and LLaVa backbone would show how much of the gap is method versus configuration.","The architecture implies that any improved single-image LMM could be converted into a video QA system with no additional training, so the practical value of the recipe may transfer to stronger image models as they are released."],"forward_implications":["Scaling frames raises accuracy monotonically on NExT-QA, from 63.5% with one frame to 70.7% with 64 frames, and the frame-by-frame design removes the context-length ceiling that limits caption-concatenation methods.","Using a question-aware distant caption adds +2.8 points over no context and +1.2 points over static distant captions, so context relevance to the question matters, not just context presence.","The same visual-plus-caption setup beats a captions-only baseline by +3.0 points when both use identical captions and the same model, supporting the paper's claim that visual and textual modalities are complementary.","Because nothing is trained, the pipeline can run on any single-image instruction-tuned LMM and any choice of frame count, making it a drop-in recipe for zero-shot video QA."],"supporting_citations":[{"why":"Contributes the question-aware caption extraction that VidCtx adopts for context captions and serves as the main open-model baseline to beat.","marker":"[13]"},{"why":"Established the captions-to-LLM pipeline that VidCtx extends by adding visual frames, and supplies the GPT-3.5/GPT-4 comparison numbers.","marker":"[11]"},{"why":"Provides the inspiration for letting the model ignore irrelevant frames and is a key comparable baseline with a trained localization component.","marker":"[8]"},{"why":"The pretrained 4-bit LLaVa-1.6-Mistral-7B model used for both captioning and answering in all VidCtx experiments.","marker":"[30]"},{"why":"Primary benchmark, source of the 70.7% top-1 accuracy and the frame-count and context ablations.","marker":"[25]"},{"why":"Second benchmark where VidCtx reports the best open-model accuracy, 67.1%.","marker":"[26]"},{"why":"Third benchmark, source of the 51.1% average accuracy and the comparison with VideoChat2.","marker":"[27]"}],"fun_headline_variants":["Distant captions boost video QA without training","Max-pooled frame votes win open-model video QA","Context-aware frame pairing lifts zero-shot VideoQA","VidCtx: image model answers video by distant frames","Training-free video QA beats prior open models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains over Q-ViD, SeViLA, LLoVi, and LangRepo come from numbers published in those papers and not from re-running them under VidCtx's exact settings; if those models were evaluated with the same frame count, quantization, caption budget, and LLaVa backbone, the margins could change.","fun_headline_variants_meta":{"raw":{"variants":["Distant captions boost video QA without training","Max-pooled frame votes win open-model video QA","Context-aware frame pairing lifts zero-shot VideoQA","VidCtx: image model answers video by distant frames","Training-free video QA beats prior open models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1405,"prompt_tokens":1050,"completion_tokens":355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":666,"tokens_out":355,"duration_ms":3918,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:26:03.674889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Q-ViD's caption-concatenation pipeline with the same LLaVa-1.6-Mistral-7B backbone, 4-bit quantization, 64-frame sampling, the same question-aware captions, and the same answer prompt; if its NExT-QA accuracy reaches or exceeds 70.7%, the paper's claimed +4.4% advantage over Q-ViD would disappear.","supporting_citations":[{"cited_title":"Question-instructed visual de- scriptions for zero-shot video answering,","cited_arxiv_id":null,"evidence_quote":"Contributes the question-aware caption extraction that VidCtx adopts for context captions and serves as the main open-model baseline to beat."},{"cited_title":"Self- chained image-language model for video localization and question answering,","cited_arxiv_id":null,"evidence_quote":"Provides the inspiration for letting the model ignore irrelevant frames and is a key comparable baseline with a trained localization component."},{"cited_title":"Llava-next: Improved reasoning, ocr, and world knowledge,","cited_arxiv_id":null,"evidence_quote":"The pretrained 4-bit LLaVa-1.6-Mistral-7B model used for both captioning and answering in all VidCtx experiments."},{"cited_title":"Next-qa: Next phase of question-answering to explaining temporal actions,","cited_arxiv_id":null,"evidence_quote":"Primary benchmark, source of the 70.7% top-1 accuracy and the frame-count and context ablations."},{"cited_title":"Intentqa: Context- aware video intent reasoning,","cited_arxiv_id":null,"evidence_quote":"Second benchmark where VidCtx reports the best open-model accuracy, 67.1%."}],"review_version":1}