Pith. sign in

REVIEW 4 major objections 4 minor 60 references

VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VideoICL claims that frozen video models can be adapted to out-of-distribution domains by similarity-ranked demonstrations and confidence-based retries, with no training.

desk verdict Solid empirical ICL for video LMMs; main claims hold, but fix the proof typo and disclose hyperparameter selection before trusting the details. read the letter →

arxiv 2412.02186 v1 pith:ZAEQ4HM2 submitted 2024-12-03 cs.CV cs.AI

Kangsan Kim , Geon Park , Youngwan Lee , Woongyeong Yeo , Sung Ju Hwang This is my paper · ORCID
classification cs.CVcs.AI
keywords videoin-contextlearningout-of-distributionunderstandinglargemultimodalmodelsconfidence-basediterativeinferencesimilarity-basedexampleselectiontokenprobabilityconfidencetraining-freeadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VideoICL claims that out-of-distribution video understanding can be improved without fine-tuning by giving a frozen video large multimodal model a small, similarity-ranked set of demonstration videos at inference time. When the model's answer is not confident enough, the framework moves to the next batch of ranked examples and repeats until the answer crosses a confidence threshold or the ranked list is exhausted. Across six benchmarks spanning question answering, classification, and captioning, the authors report an average gain of 25.6 percentage points and up to 54.6 points over zero-shot inference, with a 7-billion-parameter model beating a 72-billion-parameter zero-shot model and, on several datasets, outperforming LoRA fine-tuned models. If true, this means specialized video domains can be made usable without per-task training or larger models.

What carries the argument

Two components carry the argument. The selection score is $$ \mathrm{SQ}((t,v),(\tilde t,\tilde v)) = \$\alpha$\, \mathrm{SC}(r_t(t),r_t(\tilde t)) + (1-\$\alpha$)\, \mathrm{SC}(r_v(v),r_v(\tilde v)), $$ where $\mathrm{SC}$ is cosine similarity, $r_t$ and $r_v$ are pretrained text and video encoders, and $\alpha$ balances the two terms; the top $k$ examples by this score form an ordered pool. The retry loop then feeds $m \le k$ examples per iteration, computes confidence as $c = \min_i p_i$ over the token probabilities of the generated answer, and stops when $c > c_{\mathrm{th}}$ or the pool is exhausted, returning the highest-confidence answer. Proposition 1 states that with per-iteration accuracy $p_c$ and confidence-estimator true/false positive rates $\mathrm{TPR}$ and $\mathrm{FPR}$, the expected accuracy of the loop converges to $1/(1 + \frac{\mathrm{FPR}}{\mathrm{TPR}}\cdot\frac{1-p_c}{p_c})$ as the iteration budget grows, so a reasonably accurate confidence estimator makes iteration better than a single attempt.

What would settle it

A settling experiment is to run the framework on the crime-video classification benchmark with the confidence gate disabled (always returning the last iteration's answer) while keeping the similarity ranking, and compare against the gated version; if accuracy is unchanged, the confidence-based loop is not the source of the reported gain, and if the gate helps only on samples whose top-ranked examples share the ground-truth class, retrieval rather than confidence is doing the work.

Watch

Extended reading notes

Core claim

VideoICL's central claim is that a frozen video LMM can be steered toward out-of-distribution answers purely by which examples precede the query and by how many retries are allowed. For a given query, examples are ranked by a weighted cosine similarity between the query's text and video embeddings and each candidate's embeddings; the top $m$ examples are put in the context, and the model's answer confidence is measured as the minimum token probability of the generated response. If that confidence is below a threshold, the next $m$ ranked examples are tried, up to $k$ examples total, and the answer with the highest observed confidence is returned. The paper demonstrates the recipe on six OOD benchmarks and reports that it lifts a 7B model above zero-shot performance by an average of 25.6 percentage points, up to 54.6 points, and above a 72B zero-shot model and some LoRA fine-tuned models without any training.

Load-bearing premise

The load-bearing premise is that a small labeled pool of target-domain videos is available at test time and that ranking examples by text and video embedding similarity puts genuinely useful demonstrations at the top; without a representative pool or an informative ranking, the confidence-based retries are just reordering weak evidence.

Editorial extensions

If this is right

  • A new out-of-distribution domain can be tackled by collecting or reusing a small labeled example pool and running inference, with no gradient updates; the cost is retrieval plus a bounded number of forward passes.
  • The per-iteration batch $m$ can be chosen to fit any context window, so short-context video LMMs can still consume many demonstrations across iterations instead of compressing tokens and losing detail.
  • Early stopping on confidence makes compute adaptive: easy queries pay for one forward pass, while hard queries spend up to $k/m$ iterations, so average latency stays below the worst case.
  • If the reported gains hold, model scale is not the only route to OOD video ability: a 7B frozen model with a good example pool can beat a 72B zero-shot model, and on some tasks can beat LoRA fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the confidence gate converts a retrieval-quality problem into an adaptive-compute problem, so the same machinery should generalize to any frozen multimodal model that exposes token logits, regardless of its training recipe.
  • Editorial inference: because the final answer is the highest-confidence one rather than the last one, the framework could be strengthened by replacing the token-probability gate with a semantic-consistency score over multiple sampled answers, which is likely to be a better correctness signal on open-ended questions.
  • Editorial inference: a natural untested extension is cross-domain pools, where a single example database serves several OOD tasks at once; the current experiments keep each pool inside its target task, so the retrieval ranking has not been stressed by competing domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VideoICL, a training-free in-context learning (ICL) framework for out-of-distribution (OOD) video understanding. Given a test query, it first retrieves the top-k labeled video-text examples by a weighted cosine similarity of text and video embeddings (Eq. 1). It then iteratively feeds small batches of m retrieved examples to a frozen video large multimodal model (LMM), computes a confidence score as the minimum token probability of the generated response, and either stops or continues with the next batch until the confidence exceeds a threshold cth or all k examples are used; the final answer is the one with the highest confidence. Experiments on six OOD benchmarks (multiple-choice QA, open-ended QA, video classification, video captioning) with three backbones (LLaVA-Video-7B, Qwen2-VL-7B, Oryx-1.5-7B) show average gains of 25.6 percentage points and up to 54.6 points over zero-shot, with the 7B model sometimes outperforming a 72B zero-shot model and LoRA fine-tuned counterparts. The paper also presents a theoretical proposition (Prop. 1) intended to show that confidence-based iteration asymptotically improves accuracy.

Significance. If the empirical results hold, the work is significant: it offers a training-free, cost-effective way to adapt video LMMs to OOD domains, directly addressing the context-length bottleneck of video ICL with a simple iterative mechanism. The evaluation is broad (six datasets, four tasks, three backbones) and includes well-chosen ablations (SIMRANKONCE, RANDEXVOTE, SIMRANKVOTE) that isolate the effects of similarity-based selection and confidence-based aggregation. The paper also provides a theoretical analysis, which is a useful addition even if imperfect. However, the strength of the central empirical claim is tempered by concerns about hyperparameter selection on the test benchmarks and missing disclosure of the retrieval-weight alpha; the theoretical proof also contains a double-counting error. These issues are fixable within the manuscript's scope.

major comments (4)
  1. [Appendix C, Eq. (6)] The proof of Proposition 1 double-counts correct terminations at the final iteration. In Eq. (4), a(n) is written as the sum of c(i) for i=1 to n plus l(n-1)*pc. But c(n) already includes the probability of a correct and confident termination at iteration n (l(n-1)*pc*TPR), so adding l(n-1)*pc counts that event twice. The correct finite-n expression should be a(n) = pc*TPR*(1 - p_u^{n-1})/(1 - p_u) + p_u^{n-1}*pc. The asymptotic limit in Eq. (7) is unchanged because p_u^{n-1} vanishes, but the derivation as written is incorrect. Please correct Eq. (6) and state explicitly the condition (TPR > FPR) under which the iteration strictly improves accuracy; otherwise the theoretical guarantee in Sec. 3.4 is not established.
  2. [Sec. 4.1 and Appendix A, Table 5] The confidence threshold cth is tuned on the evaluation benchmarks, not on a held-out validation split. Table 5 reports test-set accuracy for cth in {0.1, 0.3, 0.5, 0.7, 0.9}, and the values used in the main table (cth=0.7 for Animal Kingdom, cth=0.5 for PitVQA and UCF-Crime) are chosen based on those test results, with only a cost-accuracy trade-off rationale. With no validation split, the reported gains may be partially an artifact of benchmark-driven threshold selection. Please either select cth on a validation split or demonstrate that the main results are stable across a plausible range of cth values, and report the resulting variance.
  3. [Eq. (1)] The balancing coefficient alpha between text and video similarity in Eq. (1) is never reported, nor is its selection procedure. If alpha is tuned per benchmark, the similarity-based selection advantage shown in Table 2 (e.g., Text+Video over Video-only for PitVQA) could be overstated. Please disclose the value of alpha (or the range searched) and, ideally, show that performance is robust to alpha.
  4. [Tables 1 and 2] No error bars or repeated runs are reported. Most numbers in Table 1 are single runs; only UCF-Crime and Drive&Act are averaged over splits, without standard deviation. This makes it difficult to judge whether the often modest differences (e.g., VideoICL vs SIMRANKVOTE on CapERA BLEU-4: 0.170 vs 0.165) are statistically meaningful. Please provide confidence intervals or standard deviations (at least for main baselines on representative datasets).
minor comments (4)
  1. [Sec. 4.3] The datasets paragraph says 'For open-ended QA, we utilize UCF-Crime' after already listing UCF-Crime and Drive&Act as video classification; this is a typo and should read 'For video classification'.
  2. [Sec. 4.2 / Table 1] The evaluation metric for open-ended QA (Sports-QA, PitVQA) is not specified. Please state how accuracy is computed (e.g., exact match after normalization, or external LLM judging), as this is essential for reproducibility.
  3. [Appendix E] The limitation paragraph acknowledges the reliance on a labeled example pool from the target distribution and that extremely small pools are untested. This assumption is introduced in Sec. 3.2 without much discussion; consider moving or cross-referencing this limitation in the main text to set reader expectations.
  4. [Fig. 3 caption] The phrase 'real hand-picked test samples' may raise concerns about cherry-picking; while qualitative examples are illustrative, adding a note that these are typical rather than best cases would be helpful.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: VideoICL's gains are measured against external fixed ground truth with independently ablated components; the only mild concern is a benchmark-driven cth sweep affecting the reported numbers.

  1. fitted input called prediction [Appendix A, Table 5 (confidence threshold choice); main results in Table 1]
    "While the accuracy generally increases with cth = 0.9, it also increases the cost of the entire process by performing more iterations per query on average. Therefore, we choose cth = 0.5 and 0.7 for the best trade-off between cost and accuracy. ... The values used for the main table are marked with *."

    The headline gains are generated with thresholds selected on the same test benchmarks whose scores are then reported. Table 5's caption states 'The values used for the main table are marked with *' (cth=0.7 for Animal Kingdom; cth=0.5 for PitVQA and UCF-Crime), chosen 'for the best trade-off between cost and accuracy.' The reported improvement (e.g., +54.6%p on PitVQA) is therefore the outcome of a threshold sweep on the evaluation set, so the 'prediction' partly reports the tuning objective rather than a held-out configuration. This is a partial instance of a fitted input driving the reported result. It is mild, not definitional: accuracies are still externally measured against ground truth, and ablations (VideoICL vs. SIMRANKVOTE vs.

full rationale

VideoICL's derivation chain is self-contained, and its central empirical claim is not circular. The reported accuracies (Table 1) are measured against fixed external ground-truth labels on six OOD benchmarks; no experiment defines success in terms of the method's own confidence scores or retrieval outputs. Component ablations are independently controlled: VideoICL vs. SIMRANKVOTE isolates confidence-based stopping from similarity ranking, and SIMRANKVOTE vs. RANDEXVOTE isolates similar-example selection, so the gains are not forced by a single fitted quantity. There are no self-citations: none of the cited references [1]-[59] is authored by the present authors, so no load-bearing claim is imported from prior work by the same group. The theoretical Proposition 1 (Appendix C) is a standard conditional-probability derivation from stated assumptions (constant per-iteration correctness probability pc, independent iterations, confidence TPR/FPR); the limit in Eq. (7) is derived algebraically and the guarantee is conditional on TPR > FPR, not a restatement of the empirical results. A technical flaw is noted: Eq. (6) adds l(n-1)*pc to a sum already containing c(n), double-counting the n-th iteration's correct-confident termination for finite n, although the asymptotic limit is unaffected; this is a proof bug, not circularity. The genuine weaknesses are methodological: (i) the confidence threshold cth is chosen from Table 5, an ablation on the same test benchmarks reported in Table 1 (a mild fitted-input-on-evaluation-data concern, scored 1 here, since the accuracy values are externally measured); (ii) the coefficient alpha in Eq. (1) is never reported; and (iii) Appendix E honestly concedes that the framework requires a labeled example pool from the target distribution and that extremely small pools are untested. These are disclosure and generalizability risks, not definitional circularity. Overall score 1.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new entities. It relies on standard encoders and the LMM's own logits. The main free components are the retrieval weighting alpha, the confidence threshold, and demonstration counts, all chosen on the evaluation benchmarks. The theoretical analysis adds two idealized probability assumptions that are not empirically calibrated.

free parameters (4)
  • alpha = not reported
    Balancing coefficient between text and video similarity in Eq. (1); no value is given, but all main results depend on it.
  • cth = 0.7 for MCQA, 0.5 for others
    Confidence threshold chosen from ablation on the evaluation benchmarks (Tab. 5), not on an independent validation set.
  • k = 8
    Total number of retrieved demonstrations; selected via ablation (Tab. 3) on the same test benchmarks.
  • m = 2
    Examples per iteration, set to fit within context length; implies maximum n=4 iterations.
assumptions (5)
  • ad hoc to paper The probability pc of a correct answer is constant and independent across iterations.
    Used in Proposition 1 (Sec. 3.4) to derive the asymptotic accuracy; unrealistic because later iterations use different, typically less similar, demonstrations.
  • ad hoc to paper The confidence estimator has fixed true and false positive rates independent of correctness and iteration.
    Assumed in the proof of Prop. 1; no empirical measurement of TPR and FPR is provided.
  • domain assumption A labeled example pool from the target OOD distribution is available at test time.
    Sec. 3.2 uses a database of pre-encoded examples; Appendix E acknowledges reliance on this pool and untested extremely small pools.
  • domain assumption The frozen video LMM exposes token logits for confidence computation.
    Sec. 3.3 computes min token probability from output logits; this holds for open-source models used (LLaVA-Video, Qwen2-VL, Oryx).
  • domain assumption Cosine similarity in SentenceBERT and InternVideo2 embedding space correlates with demonstration helpfulness.
    The core retrieval premise in Eq. (1); supported by ablations (Tab. 2) but not theoretically guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding." pith.science (2026). https://pith.science/paper/ZAEQ4HM2

@misc{pith2026241202186,
  author       = {Pith},
  title        = {Pith review of: VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAEQ4HM2}},
  note         = {Machine review of arXiv:2412.02186}
}
read the original abstract

Recent advancements in video large multimodal models (LMMs) have significantly improved their video understanding and reasoning capabilities. However, their performance drops on out-of-distribution (OOD) tasks that are underrepresented in training data. Traditional methods like fine-tuning on OOD datasets are impractical due to high computational costs. While In-context learning (ICL) with demonstration examples has shown promising generalization performance in language tasks and image-language tasks without fine-tuning, applying ICL to video-language tasks faces challenges due to the limited context length in Video LMMs, as videos require longer token lengths. To address these issues, we propose VideoICL, a novel video in-context learning framework for OOD tasks that introduces a similarity-based relevant example selection strategy and a confidence-based iterative inference approach. This allows to select the most relevant examples and rank them based on similarity, to be used for inference. If the generated response has low confidence, our framework selects new examples and performs inference again, iteratively refining the results until a high-confidence response is obtained. This approach improves OOD video understanding performance by extending effective context length without incurring high costs. The experimental results on multiple benchmarks demonstrate significant performance gains, especially in domain-specific scenarios, laying the groundwork for broader video comprehension applications. Code will be released at https://github.com/KangsanKim07/VideoICL

Figures

Figures reproduced from arXiv: 2412.02186 by the authors.

Figure 1
Figure 1. Motivation. Top left: Video LMMs perform poorly in out-of-distribution videos, such as crime videos. Bottom left: In-Context Learning (ICL), which is usually employed to solve this problem, is infeasible for video tasks, since the in-context demonstrations are too long. Right: VIDEOICL alleviates this problem by selecting the most relevant demonstrations (e.g., 2-shot) by similarity-based example selection, and iter… view at source ↗
Figure 2
Figure 2. Our Methodology. Given a test query Qtest consisting of a video and some text, each are embedded into a vector. Similarity￾based Example Selection: Based on the cosine similarity between the query vector and the embeddings in the database of pre-encoded examples, we retrieve top-k most similar examples. This stage takes negligible time cost since it only generates features from test samples and calculates the simila… view at source ↗
Figure 3
Figure 3. Qualitative Results. We show three real hand-picked test samples from the main benchmarks. The first and third examples are from the UCF-Crime [44] (video classification) task, and the second one is from the Sports-QA [28] (open-ended QA) task. The leftmost column shows the given question, which the vanilla (non-ICL) model makes an incorrect prediction. The second and third columns show the first and second confiden… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Most confident examples. The numbers on each bar represent the number of test samples where the corresponding it￾eration ended up having the highest confidence score. The x-axis represents the proportion of each iteration. be inadequate. Multiple iterations allow for a…
Figure 5
Figure 5. Figure 5: Qualitative result on the Animal Kingdom dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Qualitative result on the Sports-QA dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Qualitative result on the PitVQA dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Qualitative result on the UCF-Crime dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Qualitative result on the Drive&Act dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Qualitative result on the CapERA dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 30 canonical work pages

  1. [1]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024. 2

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    The Internal State of an LLM Knows When It’s Lying

    Amos Azaria and Tom Mitchell. The Internal State of an LLM Knows When It’s Lying. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023 , pages 967–976, Singapore, 2023. Association for Computational Linguistics. 3, 8

    work page 2023

  4. [4]

    What makes multimodal in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1539–1550, 2024

    Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. What makes multimodal in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1539–1550, 2024. 3

    work page 2024

  5. [5]

    Capera: Captioning events in aerial videos

    Laila Bashmal, Yakoub Bazi, Mohamad Mahmoud Al Rah- hal, Mansour Zuair, and Farid Melgani. Capera: Captioning events in aerial videos. Remote Sensing, 15(8):2139, 2023. 1, 5, 6, 8, 13

    work page 2023

  6. [6]

    In-context learn- ing with long-context models: An in-depth exploration

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learn- ing with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024. 2, 7, 8

    arXiv 2024

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    1901

  8. [8]

    Understanding and im- proving in-context learning on vision-language models

    Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, V olker Tresp, and Jindong Gu. Understanding and im- proving in-context learning on vision-language models. In ICLR 2024 Workshop on Mathematical and Empirical Un- derstanding of Foundation Models, 2024. 2, 3, 5, 6

    work page 2024

Show all 60 references
  1. [9]

    Mmict: Boosting multi- modal fine-tuning with in-context examples

    Tao Chen, Enwei Zhang, Yuting Gao, Ke Li, Xing Sun, Yan Zhang, Hui Li, and Rongrong Ji. Mmict: Boosting multi- modal fine-tuning with in-context examples. ACM Transac- tions on Multimedia Computing, Communications and Ap- plications, 2024. 3

    2024

  2. [10]

    Learning to retrieve iteratively for in-context learning

    Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, and Benjamin Van Durme. Learning to retrieve iteratively for in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7156–7168, Miam...

    2024

  3. [11]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1, 2

    2024 arXiv

  4. [12]

    Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm col- laboration

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vid- hisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm col- laboration. arXiv preprint arXiv:2402.00367, 2024. 3

    2024 arXiv

  5. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang 9 Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 8

    2024 arXiv

  6. [14]

    Aim: Let any multi-modal large language mod- els embrace efficient in-context learning

    Jun Gao, Qian Qiao, Ziqiang Cao, Zili Wang, and Wen- jie Li. Aim: Let any multi-modal large language mod- els embrace efficient in-context learning. arXiv preprint arXiv:2406.07588, 2024. 2, 3

    2024 arXiv

  7. [15]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning , pages 1321–1330. PMLR, 2017. 3

    2017

  8. [16]

    How well does GPT-4v(ision) adapt to distribution shifts? a preliminary investigation

    Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, and Kun Zhang. How well does GPT-4v(ision) adapt to distribution shifts? a preliminary investigation. In ICLR 2024 Workshop on Mathematical and Empirical Under- s...

    2024

  9. [17]

    Pitvqa: Image-grounded text embedding llm for visual question answering in pitu- itary surgery

    Runlong He, Mengya Xu, Adrito Das, Danyal Z Khan, Sophia Bano, Hani J Marcus, Danail Stoyanov, Matthew J Clarkson, and Mobarakol Islam. Pitvqa: Image-grounded text embedding llm for visual question answering in pitu- itary surgery. In International Conference on Medical Image ...

    2024

  10. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models,

  11. [19]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions. arXiv preprint arXiv:2311.05...

    2023 arXiv

  12. [20]

    Look before you leap: An exploratory study of uncertainty measurement for large language models

    Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236, 2023. 3, 4, 8

    2023 arXiv

  13. [21]

    Yixing Jiang, Jeremy Andrew Irvin, Ji Hun Wang, Muham- mad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y . Ng. Many-shot in-context learning in multimodal founda- tion models. In ICML 2024 Workshop on In-Context Learn- ing, 2024. 2

    2024

  14. [22]

    Language models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson El- hage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav F...

    2022 arXiv

  15. [23]

    Complex video rea- soning and robustness evaluation suite for video-lmms.arXiv preprint arXiv:2405.03690, 2024

    Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan, Muzammal Naseer, Federico Tombari, Fa- had Shahbaz Khan, and Salman Khan. Complex video rea- soning and robustness evaluation suite for video-lmms.arXiv preprint arXiv:2405.03690, 2024. 1, 3

    2024 arXiv

  16. [24]

    Confidence under the hood: An in- vestigation into the confidence-probability alignment in large language models

    Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, and Ali Emami. Confidence under the hood: An in- vestigation into the confidence-probability alignment in large language models. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Lingui...

    2024

  17. [25]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 8

    2023 arXiv

  18. [26]

    Mimic- it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic- it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 6

    2023 arXiv

  19. [27]

    Otter: A Multi- Modal Model with In-Context Instruction Tuning, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A Multi- Modal Model with In-Context Instruction Tuning, 2023. arXiv:2305.03726. 1, 2, 3, 5, 6

    2023 arXiv

  20. [28]

    Sports-qa: A large-scale video question answering bench- mark for complex and professional sports

    Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hos- sein Rahmani, Yulan Guo, Bernt Schiele, and Chen Chen. Sports-qa: A large-scale video question answering bench- mark for complex and professional sports. arXiv preprint arXiv:2401.01505, 2024. 1, 5, 6, 7, 13

    2024

  21. [29]

    Inference-time intervention: Elic- iting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi ´egas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Elic- iting truthful answers from a language model. In Thirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 3

    2023

  22. [30]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  23. [31]

    How to configure good in-context sequence for vi- sual question answering

    Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for vi- sual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26710–26720, 2024. 3

    2024

  24. [32]

    Teaching Models to Express Their Uncertainty in Words.Transactions on Machine Learning Research, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching Models to Express Their Uncertainty in Words.Transactions on Machine Learning Research, 2022. 3, 8

    2022

  25. [33]

    se2: Se- quential example selection for in-context learning

    Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, and Qi Zhang. se2: Se- quential example selection for in-context learning. In Find- ings of the Association for Computational Linguistics: ACL 2024, pages 5262–5284, Bangkok, Thailand, 2024. ...

    2024

  26. [34]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Ji- wen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 1, 2, 5, 6 10

    2024 arXiv

  27. [35]

    Factual Con- fidence of LLMs: on Reliability and Robustness of Current Estimators

    Mat ´eo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas M¨uller, and Lluis Marquez. Factual Con- fidence of LLMs: on Reliability and Robustness of Current Estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024

  28. [36]

    Lingoqa: Visual question answering for autonomous driv- ing

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driv- ing. In European Conference on Computer Vision , p...

    2024

  29. [37]

    Drive&act: A multi-modal dataset for fine-grained driver be- havior recognition in autonomous vehicles

    Manuel Martin, Alina Roitberg, Monica Haurilet, Matthias Horne, Simon Reiß, Michael V oit, and Rainer Stiefelhagen. Drive&act: A multi-modal dataset for fine-grained driver be- havior recognition in autonomous vehicles. In Proceedings of the IEEE/CVF International Conference o...

    2019

  30. [38]

    Animal kingdom: A large and diverse dataset for animal behavior understanding

    Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19023–19034, 2022. 1, ...

    2022

  31. [39]

    Llms know more than they show: On the intrinsic representation of llm hallucinations

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707, 2024. 8

    arXiv

  32. [40]

    Perception test: A diagnostic benchmark for multi- modal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens Continente, Larisa Markeeva, Dylan Sunil Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, An- toine Miech, Alexandre Fr´echette, Hanna Klimcza...

    2023

  33. [41]

    In-context learning with iterative demon- stration selection

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demon- stration selection. arXiv preprint arXiv:2310.09881 , 2023. 3

    2023 arXiv

  34. [42]

    Sentence-BERT: Sen- tence embeddings using Siamese BERT-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sen- tence embeddings using Siamese BERT-networks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCN...

    2019

  35. [43]

    Few-shot vqa with frozen llms: A tale of two approaches

    Igor Sterner, Weizhe Lin, Jinghong Chen, and Bill Byrne. Few-shot vqa with frozen llms: A tale of two approaches. arXiv preprint arXiv:2403.11317, 2024. 3

    2024 arXiv

  36. [44]

    Real-world anomaly detection in surveillance videos

    Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1, 5, 6, 7, 8, 13

    2018

  37. [45]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024. 5, 6, 7

    2024

  38. [46]

    GPT-4o System Card, 2024

    OpenAI Team. GPT-4o System Card, 2024. arXiv:2410.21276. 5, 6, 7

    2024 arXiv

  39. [47]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2, 5, 6

    2024 arXiv

  40. [48]

    Bayesian example selection improves in-context learning for speech, text, and visual modalities

    Siyin Wang, Chao-Han Huck Yang, Ji Wu, and Chao Zhang. Bayesian example selection improves in-context learning for speech, text, and visual modalities. arXiv preprint arXiv:2404.14716, 2024. 2, 3

    2024 arXiv

  41. [49]

    Internvideo2: Scaling video foundation mod- els for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation mod- els for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 5

    2024 arXiv

  42. [50]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 9777–9786, 2021. 3

    2021

  43. [51]

    Can LLMs express their un- certainty? an empirical evaluation of confidence elicitation in LLMs

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their un- certainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learn- ing Representations, 2024. 3, 8

    2024

  44. [52]

    From introspection to best practices: Principled anal- ysis of demonstrations in multimodal in-context learning

    Nan Xu, Fei Wang, Sheng Zhang, Hoifung Poon, and Muhao Chen. From introspection to best practices: Principled anal- ysis of demonstrations in multimodal in-context learning. arXiv preprint arXiv:2407.00902, 2024. 3

    2024 arXiv

  45. [53]

    Improving the reliability of large language models by lever- aging uncertainty-aware in-context learning

    Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. Improving the reliability of large language models by lever- aging uncertainty-aware in-context learning. arXiv preprint arXiv:2310.04782, 2023. 4

    2023 arXiv

  46. [54]

    Eliciting in-context learning in vision-language models for videos through curated data dis- tributional properties

    Keunwoo Peter Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, and Joyce Chai. Eliciting in-context learning in vision-language models for videos through curated data dis- tributional properties. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro...

    2024

  47. [55]

    Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations

    Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations. Advances in Neural Information Processing Systems, 36:58478–58507,

  48. [56]

    On the out-of-distribution generalization of multimodal large language models

    Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the out-of-distribution generalization of multimodal large language models. arXiv preprint arXiv:2402.06599, 2024. 1

    2024 arXiv

  49. [57]

    Video instruction tuning with 11 synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with 11 synthetic data. arXiv preprint arXiv:2410.02713, 2024. 1, 2, 5, 6, 8

    2024 arXiv

  50. [58]

    MMICL: Empowering vision- language model with multi-modal in-context learning

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wen- juan Han, and Baobao Chang. MMICL: Empowering vision- language model with multi-modal in-context learning. InThe Twelfth International Conference on Learning Representa- tions...

    2024

  51. [59]

    Vector-icl: In-context learning with continuous vector representations

    Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang, and Jianfeng Gao. Vector-icl: In-context learning with continuous vector representations. arXiv preprint arXiv:2410.05629, 2024. 2 12 Appendices A. Discussion on Hyperparameter Choice Animal Kingdom PitVQA UCF-Crime Avg. Z...

    2024 arXiv

  52. [2021]

    arXiv:2106.09685 [cs]. 6

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.