{"id":"90be0a45-bce9-4fc4-92ae-aad129c13986","arxiv_id":"2412.02186","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free, confidence-gated iterative in-context learning framework substantially improves out-of-distribution video understanding in QA, classification, and captioning.","lead":"VideoICL shows a video-language model a few similar solved examples and re-asks with new examples when the model is unsure, all without any training. On four video tasks, it lifts zero-shot accuracy by up to 54.6 percentage points and beats a 72B zero-shot model using a 7B backbone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Confidence-gated iteration may mainly be selecting label-distribution effects from better demonstrations; the retrieval/hyperparameter tuning protocol needs disclosure to rule out benchmark-driven selection.","rationale":"The reader's weakest_assumption focuses on the availability of a labeled target-distribution example pool and the untested extremely-small-pool regime. That is a real practical limitation, but the paper explicitly acknowledges it in Appendix E and demonstrates performance with 532 training samples on UCF-Crime, so I do not see it as the main threat to the central empirical claim. The more load-bearing concern is methodological: hyperparameters (especially cth, and the unreported alpha) appear to be selected using the evaluation benchmarks themselves, based on Appendix A Table 5 and the absence of any held-out validation description. This makes the reported 25.6%p average improvement and the specific claim that confidence-based iteration beats majority voting vulnerable to benchmark-driven selection. The proof of Proposition 1 in Appendix C also contains a double-counting mistake in Eq. (6), as identified by the reader; since the theory is not load-bearing, this does not change the verdict from conditional. The right remedy is disclosure of alpha, a validation protocol, and ideally code/error bars, which matches the reader's requested revisions. I would keep the verdict CONDITIONAL rather than ACCEPT because the central numbers need the disclosed protocol to be credible; I do not recommend REJECT because the ablations and qualitative results suggest the method is plausible and likely works as described.","tokens_in":21002,"tokens_out":1717,"duration_ms":16989,"concrete_test":"Re-run the PitVQA and UCF-Crime comparisons under a strict validation protocol: split the test set into a small validation set (e.g., 10-20%) for choosing cth (and alpha and k) and a held-out test set for final evaluation. Also run an oracle-ablated variant where cth is fixed to a single value (e.g., 0.5) across all datasets. If the held-out gains shrink dramatically or the ranking of VideoICL vs. SIMRANKVOTE flips, the headline claim of confidence-based iteration as the key mechanism needs qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that confidence-based iterative ICL, rather than the similarity-based ranking of demonstrations, drives the large OOD gains. The ablations support that similarity selection matters (Tab. 2) and that confidence-based stopping/selection beats majority voting (Tab. 1, SIMRANKVOTE vs. VideoICL). However, the comparison is confounded by hyperparameters: k=8, m=2, and cth are fixed across datasets, but cth is chosen from Appendix A Table 5 based on the same evaluation benchmarks (e.g., PitVQA and UCF-Crime use cth=0.5 because those rows were best or near-best; Animal Kingdom uses cth=0.7). With only four thresholds tested and no held-out validation split, the reported gains may be partly an artifact of benchmark-driven threshold selection. More importantly, the retrieval weight alpha in Eq. 1 is never reported; if alpha is also tuned per benchmark, the similarity selection advantage could be overstated. Additionally, the theoretical Proposition 1 proof in Appendix C has a double-counting issue: Eq. (6) adds l(n-1)*pc to the sum of c(i), but the sum already accounts for correct terminations up to iteration n-1, so the added term double-counts the n-th iteration's contribution. This does not undermine the empirical claim but weakens the theoretical guarantee that iteration strictly helps. The most load-bearing empirical concern is that with the confidence threshold and k selected on the test benchmarks, and alpha undisclosed, we cannot distinguish the proposed method's mechanism from a tuned retrieval pipeline plus early-exit on confident predictions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VideoICL, a training-free in-context learning (ICL) framework for out-of-distribution (OOD) video understanding. Given a test query, it first retrieves the top-k labeled video-text examples by a weighted cosine similarity of text and video embeddings (Eq. 1). It then iteratively feeds small batches of m retrieved examples to a frozen video large multimodal model (LMM), computes a confidence score as the minimum token probability of the generated response, and either stops or continues with the next batch until the confidence exceeds a threshold cth or all k examples are used; the final answer is the one with the highest confidence. Experiments on six OOD benchmarks (multiple-choice QA, open-ended QA, video classification, video captioning) with three backbones (LLaVA-Video-7B, Qwen2-VL-7B, Oryx-1.5-7B) show average gains of 25.6 percentage points and up to 54.6 points over zero-shot, with the 7B model sometimes outperforming a 72B zero-shot model and LoRA fine-tuned counterparts. The paper also presents a theoretical proposition (Prop. 1) intended to show that confidence-based iteration asymptotically improves accuracy.","tokens_in":21261,"tokens_out":6859,"duration_ms":65832,"significance":"If the empirical results hold, the work is significant: it offers a training-free, cost-effective way to adapt video LMMs to OOD domains, directly addressing the context-length bottleneck of video ICL with a simple iterative mechanism. The evaluation is broad (six datasets, four tasks, three backbones) and includes well-chosen ablations (SIMRANKONCE, RANDEXVOTE, SIMRANKVOTE) that isolate the effects of similarity-based selection and confidence-based aggregation. The paper also provides a theoretical analysis, which is a useful addition even if imperfect. However, the strength of the central empirical claim is tempered by concerns about hyperparameter selection on the test benchmarks and missing disclosure of the retrieval-weight alpha; the theoretical proof also contains a double-counting error. These issues are fixable within the manuscript's scope.","major_comments":[{"comment":"The proof of Proposition 1 double-counts correct terminations at the final iteration. In Eq. (4), a(n) is written as the sum of c(i) for i=1 to n plus l(n-1)*pc. But c(n) already includes the probability of a correct and confident termination at iteration n (l(n-1)*pc*TPR), so adding l(n-1)*pc counts that event twice. The correct finite-n expression should be a(n) = pc*TPR*(1 - p_u^{n-1})/(1 - p_u) + p_u^{n-1}*pc. The asymptotic limit in Eq. (7) is unchanged because p_u^{n-1} vanishes, but the derivation as written is incorrect. Please correct Eq. (6) and state explicitly the condition (TPR > FPR) under which the iteration strictly improves accuracy; otherwise the theoretical guarantee in Sec. 3.4 is not established.","section":"Appendix C, Eq. (6)"},{"comment":"The confidence threshold cth is tuned on the evaluation benchmarks, not on a held-out validation split. Table 5 reports test-set accuracy for cth in {0.1, 0.3, 0.5, 0.7, 0.9}, and the values used in the main table (cth=0.7 for Animal Kingdom, cth=0.5 for PitVQA and UCF-Crime) are chosen based on those test results, with only a cost-accuracy trade-off rationale. With no validation split, the reported gains may be partially an artifact of benchmark-driven threshold selection. Please either select cth on a validation split or demonstrate that the main results are stable across a plausible range of cth values, and report the resulting variance.","section":"Sec. 4.1 and Appendix A, Table 5"},{"comment":"The balancing coefficient alpha between text and video similarity in Eq. (1) is never reported, nor is its selection procedure. If alpha is tuned per benchmark, the similarity-based selection advantage shown in Table 2 (e.g., Text+Video over Video-only for PitVQA) could be overstated. Please disclose the value of alpha (or the range searched) and, ideally, show that performance is robust to alpha.","section":"Eq. (1)"},{"comment":"No error bars or repeated runs are reported. Most numbers in Table 1 are single runs; only UCF-Crime and Drive&Act are averaged over splits, without standard deviation. This makes it difficult to judge whether the often modest differences (e.g., VideoICL vs SIMRANKVOTE on CapERA BLEU-4: 0.170 vs 0.165) are statistically meaningful. Please provide confidence intervals or standard deviations (at least for main baselines on representative datasets).","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The datasets paragraph says 'For open-ended QA, we utilize UCF-Crime' after already listing UCF-Crime and Drive&Act as video classification; this is a typo and should read 'For video classification'.","section":"Sec. 4.3"},{"comment":"The evaluation metric for open-ended QA (Sports-QA, PitVQA) is not specified. Please state how accuracy is computed (e.g., exact match after normalization, or external LLM judging), as this is essential for reproducibility.","section":"Sec. 4.2 / Table 1"},{"comment":"The limitation paragraph acknowledges the reliance on a labeled example pool from the target distribution and that extremely small pools are untested. This assumption is introduced in Sec. 3.2 without much discussion; consider moving or cross-referencing this limitation in the main text to set reader expectations.","section":"Appendix E"},{"comment":"The phrase 'real hand-picked test samples' may raise concerns about cherry-picking; while qualitative examples are illustrative, adding a note that these are typical rather than best cases would be helpful.","section":"Fig. 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is on solid empirical ground for the broad claim that ICL with similarity-based retrieval can dramatically improve OOD video understanding. The primary issues are the hyperparameter selection protocol (cth chosen on test sets, alpha undisclosed) and the incorrect finite-n expression in the proof. These are fixable without changing the overall approach. I would also ask the authors to temper the claim of surpassing LoRA fine-tuning on UCF-Crime, since that LoRA baseline is trained for only 5 epochs with rank 32 and the comparison is not matched for compute or data budget. With these revisions, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, genuinely useful empirical paper. It applies ICL to video LMMs for OOD tasks via similarity-based retrieval plus confidence-gated iterative inference, and the main claims survive a close read.\n\nWhat's new is the combination: retrieval of video-text demonstrations ranked by text and video similarity, then iterative inference that uses a new batch of demonstrations until confidence exceeds a threshold. Each ingredient is known, but the video-specific constraint of limited context length makes the iterative aspect practical and the results bear that out. The evaluation is broad: six datasets across QA, classification, and captioning, three backbones, and ablations that cleanly separate similarity selection, iteration count, and confidence estimator. The gains are large (25.6 points average, up to 54.6 on PitVQA) and consistent. The ablation showing confidence selection beats majority voting is a genuinely informative result.\n\nThe soft spots are real but fixable. First, the proof of Proposition 1 in Appendix C double-counts the terminal iteration: Eq. (6) adds l(n-1)*pc to a sum that already includes c(n). The limit in Eq. (7) is correct because the extra term vanishes, so the proposition survives, but the derivation needs correction. Second, cth and k are selected on the evaluation benchmarks (Tables 3 and 5), with no held-out validation. That is benchmark tuning and should either be disclosed or avoided via a validation split. Third, alpha in Eq. (1) is never reported, so the retrieval score is not fully specified. Fourth, no error bars and no released code yet, which limits reproducibility.\n\nI want to push back slightly on the stress-test's sharpest version of the threshold-tuning concern. The gains on PitVQA and UCF-Crime are still very large at cth=0.1, so threshold choice cannot explain the central finding. The tuning issue matters for the claimed operating point, not for the core result.\n\nWho this is for: anyone working on video LMMs, ICL, or OOD generalization in multimodal settings. It is a practical, training-free method with solid evidence. My recommendation: send it to peer review. Ask the authors to fix the proof, report alpha and the threshold-selection procedure, and ideally release code and error bars. With those changes, this is an accept.","headline":"Solid empirical ICL for video LMMs; main claims hold, but fix the proof typo and disclose hyperparameter selection before trusting the details.","tokens_in":21847,"tokens_out":4070,"would_cite":true,"duration_ms":41277,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VideoICL claims that frozen video models can be adapted to out-of-distribution domains by similarity-ranked demonstrations and confidence-based retries, with no training.","keywords":["video in-context learning","out-of-distribution video understanding","video large multimodal models","confidence-based iterative inference","similarity-based example selection","token probability confidence","training-free adaptation"],"falsifier":"A settling experiment is to run the framework on the crime-video classification benchmark with the confidence gate disabled (always returning the last iteration's answer) while keeping the similarity ranking, and compare against the gated version; if accuracy is unchanged, the confidence-based loop is not the source of the reported gain, and if the gate helps only on samples whose top-ranked examples share the ground-truth class, retrieval rather than confidence is doing the work.","tokens_in":20748,"feed_emoji":"🎬","tokens_out":14014,"duration_ms":120524,"temperature":0.7,"pith_summary":"VideoICL claims that out-of-distribution video understanding can be improved without fine-tuning by giving a frozen video large multimodal model a small, similarity-ranked set of demonstration videos at inference time. When the model's answer is not confident enough, the framework moves to the next batch of ranked examples and repeats until the answer crosses a confidence threshold or the ranked list is exhausted. Across six benchmarks spanning question answering, classification, and captioning, the authors report an average gain of 25.6 percentage points and up to 54.6 points over zero-shot inference, with a 7-billion-parameter model beating a 72-billion-parameter zero-shot model and, on several datasets, outperforming LoRA fine-tuned models. If true, this means specialized video domains can be made usable without per-task training or larger models.","feed_headline":"Frozen video models gain up to 54.6 points on OOD tasks","feed_subtitle":"No fine-tuning: ranked video examples plus confidence retries lift frozen 7B models above zero-shot on six OOD tasks","key_machinery":"Two components carry the argument. The selection score is\n$$\n\\mathrm{SQ}((t,v),(\\tilde t,\\tilde v)) = \\$\\alpha$\\, \\mathrm{SC}(r_t(t),r_t(\\tilde t)) + (1-\\$\\alpha$)\\, \\mathrm{SC}(r_v(v),r_v(\\tilde v)),\n$$\nwhere $\\mathrm{SC}$ is cosine similarity, $r_t$ and $r_v$ are pretrained text and video encoders, and $\\alpha$ balances the two terms; the top $k$ examples by this score form an ordered pool. The retry loop then feeds $m \\le k$ examples per iteration, computes confidence as $c = \\min_i p_i$ over the token probabilities of the generated answer, and stops when $c > c_{\\mathrm{th}}$ or the pool is exhausted, returning the highest-confidence answer. Proposition 1 states that with per-iteration accuracy $p_c$ and confidence-estimator true/false positive rates $\\mathrm{TPR}$ and $\\mathrm{FPR}$, the expected accuracy of the loop converges to $1/(1 + \\frac{\\mathrm{FPR}}{\\mathrm{TPR}}\\cdot\\frac{1-p_c}{p_c})$ as the iteration budget grows, so a reasonably accurate confidence estimator makes iteration better than a single attempt.","core_discovery":"VideoICL's central claim is that a frozen video LMM can be steered toward out-of-distribution answers purely by which examples precede the query and by how many retries are allowed. For a given query, examples are ranked by a weighted cosine similarity between the query's text and video embeddings and each candidate's embeddings; the top $m$ examples are put in the context, and the model's answer confidence is measured as the minimum token probability of the generated response. If that confidence is below a threshold, the next $m$ ranked examples are tried, up to $k$ examples total, and the answer with the highest observed confidence is returned. The paper demonstrates the recipe on six OOD benchmarks and reports that it lifts a 7B model above zero-shot performance by an average of 25.6 percentage points, up to 54.6 points, and above a 72B zero-shot model and some LoRA fine-tuned models without any training.","pith_inferences":["Editorial inference: the confidence gate converts a retrieval-quality problem into an adaptive-compute problem, so the same machinery should generalize to any frozen multimodal model that exposes token logits, regardless of its training recipe.","Editorial inference: because the final answer is the highest-confidence one rather than the last one, the framework could be strengthened by replacing the token-probability gate with a semantic-consistency score over multiple sampled answers, which is likely to be a better correctness signal on open-ended questions.","Editorial inference: a natural untested extension is cross-domain pools, where a single example database serves several OOD tasks at once; the current experiments keep each pool inside its target task, so the retrieval ranking has not been stressed by competing domains."],"forward_implications":["A new out-of-distribution domain can be tackled by collecting or reusing a small labeled example pool and running inference, with no gradient updates; the cost is retrieval plus a bounded number of forward passes.","The per-iteration batch $m$ can be chosen to fit any context window, so short-context video LMMs can still consume many demonstrations across iterations instead of compressing tokens and losing detail.","Early stopping on confidence makes compute adaptive: easy queries pay for one forward pass, while hard queries spend up to $k/m$ iterations, so average latency stays below the worst case.","If the reported gains hold, model scale is not the only route to OOD video ability: a 7B frozen model with a good example pool can beat a 72B zero-shot model, and on some tasks can beat LoRA fine-tuning."],"supporting_citations":[{"why":"the frozen video LMM backbone that fixes the context-length constraint and supplies all main results.","marker":"[57]"},{"why":"the surgical-video open-ended QA benchmark where the largest reported gain occurs.","marker":"[17]"},{"why":"the crime-video classification benchmark used in the main comparison and in the LoRA comparison.","marker":"[44]"},{"why":"the sports-video open-ended QA benchmark used for main results and qualitative retry examples.","marker":"[28]"},{"why":"the animal-video multiple-choice QA benchmark used in main results and ablations.","marker":"[38]"},{"why":"the text encoder used to compute the text-similarity term of the selection score.","marker":"[42]"},{"why":"the video encoder used to compute the video-similarity term of the selection score.","marker":"[49]"},{"why":"the prior work supplying the minimum-token-probability confidence measure that drives the retry loop.","marker":"[20]"},{"why":"the in-context-trained baseline that the training-free framework is compared against.","marker":"[27]"}],"fun_headline_variants":["Training-free iterative ICL boosts OOD video understanding by up to 54.6 points","Confidence-based retries lift frozen video LMMs above zero-shot on OOD tasks","Iterative example re-ranking gains up to 54.6 points on OOD video tasks","Frozen 7B model outperforms 72B zero-shot via confidence retries","No training, just iterative ICL: OOD video understanding improves up to 54.6 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a small labeled pool of target-domain videos is available at test time and that ranking examples by text and video embedding similarity puts genuinely useful demonstrations at the top; without a representative pool or an informative ranking, the confidence-based retries are just reordering weak evidence.","fun_headline_variants_meta":{"raw":{"variants":["Training-free iterative ICL boosts OOD video understanding by up to 54.6 points","Confidence-based retries lift frozen video LMMs above zero-shot on OOD tasks","Iterative example re-ranking gains up to 54.6 points on OOD video tasks","Frozen 7B model outperforms 72B zero-shot via confidence retries","No training, just iterative ICL: OOD video understanding improves up to 54.6 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2808,"prompt_tokens":995,"completion_tokens":1813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1696}},"tokens_in":611,"tokens_out":1813,"duration_ms":12329,"temperature":1.0,"reasoning_tokens":1696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:45:17.915241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A settling experiment is to run the framework on the crime-video classification benchmark with the confidence gate disabled (always returning the last iteration's answer) while keeping the similarity ranking, and compare against the gated version; if accuracy is unchanged, the confidence-based loop is not the source of the reported gain, and if the gate helps only on samples whose top-ranked examples share the ground-truth class, retrieval rather than confidence is doing the work.","supporting_citations":[{"cited_title":"Pitvqa: Image-grounded text embedding llm for visual question answering in pitu- itary surgery","cited_arxiv_id":null,"evidence_quote":"the surgical-video open-ended QA benchmark where the largest reported gain occurs."},{"cited_title":"Real-world anomaly detection in surveillance videos","cited_arxiv_id":null,"evidence_quote":"the crime-video classification benchmark used in the main comparison and in the LoRA comparison."},{"cited_title":"Animal kingdom: A large and diverse dataset for animal behavior understanding","cited_arxiv_id":null,"evidence_quote":"the animal-video multiple-choice QA benchmark used in main results and ablations."},{"cited_title":"Sentence-BERT: Sen- tence embeddings using Siamese BERT-networks","cited_arxiv_id":null,"evidence_quote":"the text encoder used to compute the text-similarity term of the selection score."}],"review_version":1}