Pith. sign in

REVIEW 5 major objections 4 minor 29 references

Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video-LLMs hallucinate answers to out-of-scope questions because nothing in their training teaches refusal; alignment on the UVQA dataset installs that skill, lifting answerability F1 from zero to roughly 0.65.

desk verdict A useful alignment-for-answerability package with a new dataset and metrics; the empirical gains are consistent, but the causal claim about video understanding is under-supported and the OOD evidence is too thin to carry it. read the letter →

arxiv 2507.04976 v1 pith:HIDXX4DA submitted 2025-07-07 cs.CV cs.CL

Eunseop Yoon , Hee Suk Yoon , Mark A. Hasegawa-Johnson , Chang D. Yoo This is my paper
classification cs.CVcs.CL
keywords videolargelanguagemodelsanswerabilityunanswerablequestionsrefusalbehavioralignmenthallucinationUVQAdatasetdirectpreferenceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that video large language models answer unanswerable questions not because they cannot see what the video contains, but because their training corpora never include questions that lie outside the video's information, so the models have no learned policy for refusing. It formalizes this as 'alignment for answerability': score a response as correct only when an answerable question gets an accurate answer and an unanswerable question gets a refusal with correct reasoning. With the synthetic UVQA dataset (30k training questions built by swapping objects, relations, or attributes in existing video descriptions), supervised fine-tuning or DPO alignment lifts three open models from an answerability F1 of 0.00 to roughly 0.65 while only mildly reducing performance on answerable questions, and the behavior transfers to videos from an unrelated dataset. A sympathetic reader should care because silent hallucination on out-of-scope questions is exactly the failure that undermines trust in video QA systems in real deployment.

What carries the argument

The load-bearing object is the response-typing function $t(y)$, which classifies each model answer as correct, wrong, unanswerable with correct reasoning, or unanswerable with wrong reasoning, joined with the scoring identity $s(v,x,y) = 1$ iff $k(v,x)\cdot t(y) = 1$, where $k(v,x)$ records whether the video contains the information the question asks for. This identity converts answerability from an aspiration into a per-example label: a model is rewarded only when it answers correctly on answerable questions and refuses with accurate reasoning on unanswerable ones, and the three alignment metrics (excessive refusal, permissiveness, discretion) decompose the resulting behavior change so a reader can tell improved discernment apart from blanket over-caution. The supporting machinery is the UVQA pipeline, which swaps one element of a scene-graph triplet (object or relation, from MOMA-LRG) or one adjective category (from DiDeMo) inside a video description, then prompts an external LLM to produce an unanswerable question and a refusal rationale; within-category swaps keep the altered descriptions natural.

What would settle it

Re-annotate a random sample of the UVQA evaluation questions against the original videos with annotators who never see the pipeline's labels, and measure how often a question labeled unanswerable is actually answerable from incidental video content; a nontrivial error rate would mean the reported refusal gains partly reflect label noise rather than true answerability learning. A second check: give the aligned model questions whose answerability is fixed by independent human annotation, and see whether its refusals track true answerability or merely the question's phrasing and swapped-entity structure.

Watch

Extended reading notes

Core claim

The central claim is that answerability is a trainable behavior that scale and instruction-following do not deliver by themselves: the paper reports that current Video-LLMs, including a 72B model, score essentially zero on its unanswerable-question benchmark even while improving on standard answerable benchmarks, a contrast it interprets as evidence that refusal capability is missing from the training distribution rather than from the perception stack. Alignment for answerability installs that capability by training the model to satisfy $k(v,x)\cdot t(y) = 1$ — answer correctly when the video supports the question, and answer with 'unanswerable' plus correct reasoning when it does not. The paper reports that aligned models reach an answerability F1 around 0.65 and raise balanced accuracy from about 0.23 to about 0.5 across three open backbones (Video-LLaVA, VideoChat2, and VLM-RLAIF), with DPO delivering the refusal behavior more softly than SFT, and it documents the acknowledged tradeoff that alignment also raises the excessive-refusal rate on questions the model previously handled.

Load-bearing premise

The load-bearing premise is that swapping an object, relation, or attribute in a written video description reliably yields a question the actual video cannot answer; if a description overlooks something visible in the video, or the generating LLM writes a question answerable from incidental content, the ground-truth labels are wrong, and both the 30k-sample training set and the human-filtered evaluation inherit that error.

Editorial extensions

If this is right

  • Deployed Video-LLMs would stop silently fabricating answers: on out-of-scope questions the aligned models answer 'unanswerable' and state why, converting the dominant failure mode from confident hallucination into a usable signal.
  • The refusal skill transfers to videos outside the training distribution (the MSR-VTT evaluation), so the alignment recipe can be applied to new backbones and new video sources without recollecting per-video data.
  • The choice between SFT and DPO becomes a practical dial: SFT maximizes discretion to detect unanswerable questions, while DPO keeps more of the original answering behavior with lower excessive refusal, so applications can pick an alignment point from the reported Pareto tradeoff.
  • Prompting and inference-time existence-question splitting do not substitute for training: an explicit answerability prompt barely moves the models, and splitting a question into existence checks both underperforms trained alignment and multiplies evaluation cost several-fold.
  • Standard video QA benchmarks miss the refusal deficit entirely, so reporting answerability-aware accuracy alongside conventional accuracy would reveal capabilities that current leaderboards hide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The scoring identity is modality-blind: the same response-typing and reward shaping could be applied to image-only, audio, or embodied multimodal LLMs, making this a template for answerability alignment rather than a video-specific fix.
  • Synthetic label noise is likely the binding constraint on the reported ceiling; auditing the training set with an independent answerability oracle and filtering or reweighting disputed samples could plausibly push refusal F1 well past the observed 0.65.
  • A natural unstated extension is uncertainty-conditioned refusal — abstaining when the model's own confidence is low rather than when the question structure matches a training pattern — which the paper's metrics could directly evaluate as a way to reduce the excessive-refusal tradeoff it acknowledges.
  • Because the two-dimensional evaluation (answerable plus unanswerable halves) is cheap to construct, the paper's framework also suggests a benchmark convention: report refusal-aware accuracy as answerability detection multiplied by answer correctness, making different video QA systems comparable on this axis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper addresses the tendency of Video-LLMs to answer questions that are unanswerable given the video content, by proposing an 'alignment for answerability' framework. The authors formalize a scoring function distinguishing correct answers, wrong answers, and two types of unanswerability responses, and introduce metrics for excessive refusal, permissiveness, discretion, and an overall alignment score. They construct UVQA, a dataset of 30k synthetic unanswerable question-answer pairs generated by altering scene-graph descriptions (objects, relations, attributes) and prompting GPT-4, plus 300 human-filtered evaluation samples. They apply SFT and DPO to Video-LLaVA, VideoChat2, and VLM-RLAIF, reporting consistent improvements in answerability F1, alignment scores, accuracy, and LLM score, with human evaluation on 100 samples and an out-of-distribution evaluation on 100 MSR-VTT samples. The paper also includes a POPE-style prompting baseline and a Pareto analysis of the answerable/unanswerable trade-off.

Significance. The problem is timely and the proposed framework is a useful formalization with practical metrics and a released dataset. The strongest evidence for the central claim is the consistent improvement across three base models, the human evaluation, and the out-of-distribution test. The release of code and data is a notable strength. However, the causal interpretation—that failures reflect missing training rather than limited video understanding—is only indirectly supported, and the out-of-distribution evidence is too coarse to rule out distribution-specific cue fitting.

major comments (5)
  1. [Appendix A.12, Table 6] The out-of-distribution evaluation consists of only 100 human-annotated MSR-VTT samples and reports only aggregate metrics (Table 6). Because the in-distribution results (Table 2) show that attribute-related unanswerability is consistently harder than object- and relation-related types, and the UVQA generation pipeline is template-based, the aggregate OOD numbers cannot distinguish a general answerability skill from pattern-matching on question forms. To support the generalization claim, the authors should provide a per-category breakdown of the OOD set, report inter-annotator agreement, and ideally increase the sample size; without this, the claim that alignment transfers beyond the training distribution is not established.
  2. [Section 4.2, Eqs. (5)-(7), Table 1] The unaligned baseline is reported with Sdisc = 0.33 for all three models, yet Sdisc is defined as (N11+N12)/(N10+N11+N12), a transition-based measure between M and M'. With M'=M for the unaligned baseline, no transitions can occur, so Sdisc should be 0, matching the reported Sex-ref = 0 and Spermis = 0. The reported value suggests either the metric is computed as an absolute rate (which would change its interpretation) or the category definitions are not what the text states. This discrepancy affects Salign and the comparison between aligned and unaligned models; please clarify and correct the definitions or the reported values.
  3. [Abstract and Section 1] The statement that failures are 'not necessarily due to a lack of video understanding, but because they have not been trained to identify and refuse such questions' is presented as a main finding, but the experimental design does not measure video understanding independently of answerability behavior. The POPE-style experiment (Section 6) uses an external LLM to generate existence subquestions and GPT-4o to verify the responses, so improvements there do not isolate the video understanding of the target model. To support the causal claim, the authors should add a controlled test, e.g., comparing alignment performance on questions whose unanswerability depends on absent entities versus questions that are answerable but require the same perceptual evidence, or explicitly relax the causal language in the abstract to describe an observed improvement.
  4. [Section 4.3 and Section 5.1] The UVQA training set (30k samples) is filtered only automatically, despite the pipeline's core assumption that altering a scene-graph element yields an unanswerable question. If the original video contains the replacement object, or the LLM-generated question can be answered from other video content, the training label is wrong. Since the model is trained on these labels, the measured gains on the human-filtered evaluation set could be achieved despite substantial training noise, but the paper does not quantify this. Please report a human audit on a random sample of the training set, or at minimum discuss the estimated label error rate and its effect on alignment.
  5. [Section 4.2 and Appendix A.5] The in-distribution evaluation relies on GPT-4o to judge whether a model's refusal reasoning 'matches' the GPT-4-turbo-generated ground truth. Because label generation and evaluation both come from the same language-model family, part of the measured improvement may reflect stylistic mimicry rather than robust answerability reasoning. The human evaluation in Appendix A.11 covers only 100 samples and does not report agreement with the GPT-4 judgments; please provide a comparison of human and GPT-4 type(y) classifications on a common sample set.
minor comments (4)
  1. [Global] There are several typos, including 'Langauge' in the Section 2 header and 'failes to detect' in the description of Figure 4; please copyedit the manuscript.
  2. [Figure 2 and Section 4.2] The category numbers N1–N18 are defined only via Figure 2, but the figure is not included in the main text; please add a complete textual definition of all 18 categories so the metrics can be verified without relying on a figure.
  3. [Table 1] The 'Answerability F1' metric is reported without a definition of the positive class or averaging scheme (macro/micro); please specify how the F1 is computed.
  4. [Appendix A.11] The human evaluation reports average scores over 100 samples but no variance or inter-annotator agreement; please add standard deviations or a similar measure of reliability.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: UVQA’s definition of unanswerability and its GPT-4-written y_gt are both the training target and the in-distribution scoring reference; independent OOD and human checks keep the central claim from collapsing.

  1. self definitional [Section 4.3, Eq. 9 (UVQA generation pipeline)]
    "Given a dataset consisting of video-description pairs (v, d), we generate an altered description d′, which provides an incorrect scene description of v ... we then prompt an external mature LLM (gpt-4-turbo-2024-04-09) to generate a question x based on d′ along with an answer ygt, where ygt includes an unanswerable indicator ... resulting in an unanswerable question x for the video v ... (x,y gt) =LLM(d ′,c),wherek(v,x) =−1."

    The paper’s formal definition of unanswerability, k(v,x) = −1, is installed here by construction: any question generated from the altered description d′ is declared unanswerable for v, with no independent check at training time beyond automatic filtering. The same construction defines both the 30k UVQA training samples and the 300-sample in-distribution evaluation set (Section 5.1). Consequently, the in-distribution result—that models fail before alignment and improve after UVQA alignment—partly measures agreement with the generation pipeline’s own label convention rather than a video-grounded property. Human filtering removes obvious mislabels but preserves the shared question-form distribution, so the improvement is partially self-confirming.

  2. fitted input called prediction [Section 4.1 (Eq. 1) and Section 5.3 / Appendix A.5 (evaluation prompts)]
    "type(y) = unanswerable c when the response y contains unanswerable indicators and the reasoning for why it is unanswerable is consistent with the ground truth ygt. ... we include the LLMscore, which assesses the quality of the model-generated response y in comparison to the ground truth label ygt. This rating, assigned by GPT-4, is on a scale from 0 to 5."

    The same GPT-4-family process that writes y_gt in Eq. 9 also supplies the reference against which GPT-4 judges the model at evaluation: the ‘unanswerable_c’ category requires the model’s reasoning to be consistent with that GPT-4-written y_gt, and the LLMscore is a GPT-4 rating of y against y_gt. Since SFT/DPO train the model to emit those very y_gt targets, the evaluation rewards stylistic and content agreement with the label generator. Part of the reported alignment gain is therefore a fit to the training/evaluation reference, not independent evidence of a general refusal skill. This is partial: answerability F1 does not require reasoning match, and the human-annotated OOD labels break the generator identity.

full rationale

The paper’s central in-distribution evidence is not fully self-contained: UVQA’s ground-truth unanswerability label is assigned by construction in Eq. 9 (replace a description element, then have GPT-4 write both the question and the reasoning), and the same GPT-4-written y_gt is the reference used by GPT-4 in Appendix A.5 to decide whether a model’s refusal reasoning is ‘unanswerable_c’ and to assign LLMscore. Training with SFT/DPO on those labels therefore improves the very comparison the in-distribution metrics reward; part of the reported gain is fit to the label generator’s style rather than an independently verified answerability skill. However, the paper does break the loop at three points: (1) the 300-sample in-distribution eval set is human-filtered; (2) Appendix A.12 uses a separate MSR-VTT video source with human-written unanswerable QA pairs, where aligned models still outperform unaligned models; (3) Appendix A.11 reports human ratings of model outputs. Those independent checks keep the central claim from reducing entirely to the synthetic pipeline. The acknowledged excessive-refusal trade-off (Appendix A.2) is a limitation, not a circularity. No load-bearing self-citation or imported uniqueness theorem was found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numeric free parameters are fitted to the evaluation data; training uses standard hyperparameters (lr 1e-6, batch size 128). The paper introduces a new dataset and metrics rather than new physical or mechanistic entities. The key assumptions are about the reliability of synthetic description-alteration labels, GPT-4-based evaluation, and the binary definition of answerability.

assumptions (5)
  • domain assumption The source video-description datasets (MOMA-LRG, DiDeMo) are accurate enough that replacing an object, relation, or attribute always yields a description that contradicts the video content.
    Section 4.3 builds all UVQA unanswerability on this replacement guarantee; if the original description misses objects present in the video, the 'unanswerable' question may actually be answerable.
  • domain assumption LLM-generated ground truth reasoning y_gt is a valid and complete explanation of why each question is unanswerable.
    Equation (9) and automatic filtering accept GPT-4 generated labels for 30k training samples; only the 300-sample evaluation set is human-filtered.
  • domain assumption GPT-4 classification of model responses into correct/wrong/unanswerable types is a reliable proxy for human judgment.
    Appendix A.5 uses gpt-4o prompts to compute type(y) for all alignment metrics; human evaluation covers only 100 samples.
  • domain assumption The binary answerability function k(v,x) is well-defined, ignoring partial answerability or questions answerable from general knowledge rather than video content.
    Equation (2) defines k(v,x) as exactly +1 or -1; this simplification underlies all metrics and dataset labels.
  • domain assumption Failure to refuse unanswerable questions is due to lack of training rather than deficient video perception.
    The paper uses this causal claim to motivate alignment; evidence is indirect, including Figure 1c, scaling results in Figure 1b, and prompt experiments in Appendix A.9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models." pith.science (2026). https://pith.science/paper/HIDXX4DA

@misc{pith2026250704976,
  author       = {Pith},
  title        = {Pith review of: Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HIDXX4DA}},
  note         = {Machine review of arXiv:2507.04976}
}
read the original abstract

In the broader context of deep learning, Multimodal Large Language Models have achieved significant breakthroughs by leveraging powerful Large Language Models as a backbone to align different modalities into the language space. A prime exemplification is the development of Video Large Language Models (Video-LLMs). While numerous advancements have been proposed to enhance the video understanding capabilities of these models, they are predominantly trained on questions generated directly from video content. However, in real-world scenarios, users often pose questions that extend beyond the informational scope of the video, highlighting the need for Video-LLMs to assess the relevance of the question. We demonstrate that even the best-performing Video-LLMs fail to reject unfit questions-not necessarily due to a lack of video understanding, but because they have not been trained to identify and refuse such questions. To address this limitation, we propose alignment for answerability, a framework that equips Video-LLMs with the ability to evaluate the relevance of a question based on the input video and appropriately decline to answer when the question exceeds the scope of the video, as well as an evaluation framework with a comprehensive set of metrics designed to measure model behavior before and after alignment. Furthermore, we present a pipeline for creating a dataset specifically tailored for alignment for answerability, leveraging existing video-description paired datasets.

Figures

Figures reproduced from arXiv: 2507.04976 by the authors.

Figure 1
Figure 1. Limitations of Current Video-LLMs. (a) While Video-LLMs demonstrate steady improvements on traditional video understanding and QA benchmarks, their performance on our unanswerable question evaluation benchmark remains poor, with no meaningful progress observed across models. (b) Scaling the model size (7B → 72B) leads to improvements on traditional answerable QA benchmark, while performance on our unanswerable quest… view at source ↗
Figure 2
Figure 2. All possible scenarios of model response type changes between the pre-aligned model (M) and post-aligned model (M′ ). (a) shows cases where the question x is answerable based on the input video v (i.e., k(v, x) = 1), while (b) depicts the cases where the question x is unanswerable given the input video v (i.e., k(v, x) = −1). Note that for (a) Answerable Questions, unanswerable responses are grouped as type(y) = una… view at source ↗
Figure 3
Figure 3. Dataset Creation for Alignment for Answerability. The process begins with a video￾description paired dataset (v, d). A Description Altering Process T modifies the original description d by applying a change c, producing an altered description d ′ . The altered description d ′ and the modification c are then input into a large language model (LLM) to generate an unanswerable question x and the corresponding reasoning… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Examples of model predictions from the unaligned model, the model aligned using Supervised Fine-Tuning (SFT), and the model aligned using Direct Preference Optimization (DPO), all based on VLM-RLAIF (Ahn et al., 2024). The examples illustrate (a) Object-related (O), (b…
Figure 5
Figure 5. Figure 5: Performance Comparison Between Unaligned, POPE-Style, and Aligned Models (Ours). (a) & (b) The POPE-Style method demonstrates improved performance over the unaligned baseline on both the alignment metric and overall accuracy. However, it falls short when compared to th…
Figure 6
Figure 6. Figure 6: Prompt used to instruct UVQA on object-related O questions [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Prompt used to instruct UVQA on relation-related R questions [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Prompt used to instruct UVQA on attribute-related A questions 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Examples of Automatically Filtered Data in the UVQA Training Set. Two main cases were filtered: (a) Generated Descriptions with Semantic Similarity and (b) Generated Descriptions with Grammatical Errors [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Illustration of the interface used for human filtering in the UVQA test set. The interface displays video frames, the original description d, the altered description d ′ , and the generated QA pair (x, ygt). Reviewers manually assess the quality of each QA pair based …
Figure 11
Figure 11. Figure 11: Examples of object-related unanswerable question-answer pair from our generated UVQA Dataset [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Examples of relation-related unanswerable question-answer pair from our generated UVQA Dataset 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Examples of attribute-related unanswerable question-answer pair from our generated UVQA Dataset A.5 PROMPT USED FOR EVALUATION To determine the type(y) of the model’s response, we use a GPT-4 model (gpt-4o-2024-05-13) with the prompts shown in [PITH_FULL_IMAGE:figure…
Figure 14
Figure 14. Figure 14: Prompt used for Evaluation: (a) Evaluation prompt for answerable dataset, and (b) Evaluation prompt for our unanswerable dataset. A.6 POPE-STYLE UNANSWERABILITY DETECTION In Section 6, we experiment if we can detect unanswerable questions by reformulating them into a …
Figure 15
Figure 15. Figure 15: Prompt used to generate POPE-Style question [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Examples of Generated Existence-based Question Set A.7 ADDITIONAL EXAMPLES OF MODEL PREDICTIONS In [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Additional examples of model prediction from the unaligned model, the model aligned [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Additional examples of model prediction from the unaligned model, the model aligned [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Pareto front visualization using the VLM-RLAIF model (Ahn et al., 2024). [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: (a) Interface provided to human annotators for evaluation. (b) Comparison of scores [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Interface provided to human annotators for collecting out-of-distribution evaluation [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 14 canonical work pages

  1. [1]

    Figure 14:Prompt used for Evaluation:(a) Evaluation prompt for answerable dataset, and (b) Evaluation prompt for our unanswerable dataset

    This score assesses the quality of the model-generated response y against the ground truth ygt, using a rating scale from 0 to 5, as outlined in the evaluation prompt. Figure 14:Prompt used for Evaluation:(a) Evaluation prompt for answerable dataset, and (b) Evaluation prompt for our unanswerable dataset. A.6 POPE-STYLEUNANSWERABILITYDETECTION In Section ...

    work page 2025

  2. [2]

    URL https://openreview.net/forum?id= f1UL4wNlw6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris H...

    work page 1901

  3. [3]

    A.3 ETHICSSTATEMENT In our study, we utilize Large Language Models (LLM) to generate our UVQA dataset and evaluate video-LLMs, which may result in unintended outcomes

    that achieve a better balance, reducing the excessive refusal score without compromising the model’s overall accuracy. A.3 ETHICSSTATEMENT In our study, we utilize Large Language Models (LLM) to generate our UVQA dataset and evaluate video-LLMs, which may result in unintended outcomes. However, during the human filtering process of generating the UVQA eva...

    work page 2025

  4. [4]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  5. [5]

    Unanswerable visual question answering.arXiv preprint arXiv:2310.10942,

    11 Published as a conference paper at ICLR 2025 Yanyang Guo, Fangkai Jiao, Zhiqi Shen, Liqiang Nie, and Mohan Kankanhalli. Unanswerable visual question answering.arXiv preprint arXiv:2310.10942,

    arXiv 2025

  6. [11]

    URL https://aclanthology

    Association for Computational Linguistics. URL https://aclanthology. org/2024.acl-long.679. Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypernetworks. InInternational Conference on Learning Representations,

    work page 2024

  7. [12]

    Language models can see: Plugging visual controls in text generation.arXiv preprint arXiv:2205.02655,

    Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation.arXiv preprint arXiv:2205.02655,

    arXiv

  8. [13]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525,

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525,

    arXiv

Show all 29 references
  1. [14]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  2. [15]

    Know your limits: A survey of abstention in large language models.arXiv preprint arXiv:2407.18418,

    Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. Know your limits: A survey of abstention in large language models.arXiv preprint arXiv:2407.18418,

    arXiv

  3. [16]

    Reliable visual question answering: Abstain rather than answer incorrectly

    13 Published as a conference paper at ICLR 2025 Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. InEuropean Conference on Computer ...

    2025

  4. [18]

    TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback. In Lun-Wei Ku, Andre Martins, and V...

    2024 doi

  5. [19]

    doi: 10.18653/v1/2022.emnlp-main.280

    Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.280. URL https://aclanthology.org/2022.emnlp-main.280/. Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Yoon, Junyeong Kim, and Chang Yoo. HEAR: Hearing enhanced audio response for video-grounded dialogue. ...

    2022 doi

  6. [20]

    doi: 10.18653/v1/2023.findings-emnlp.797

    Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.797. URL https://aclanthology.org/2023. findings-emnlp.797/. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: To...

    2023

  7. [22]

    doi: 10.18653/ v1/2023.emnlp-demo.49

    Association for Computational Linguistics. doi: 10.18653/ v1/2023.emnlp-demo.49. URLhttps://aclanthology.org/2023.emnlp-demo.49. 14 Published as a conference paper at ICLR 2025 Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chu...

    2023 arXiv

  8. [23]

    URL https://openreview.net/forum? id=1tZbq88f27. 15 Published as a conference paper at ICLR 2025 A APPENDIX A.1 BROADERIMPACT This paper highlights the importance of aligning Video Large Language Models (Video-LLMs) to handle unanswerable questions, a critical capability often...

    2025

  9. [25]

    (2023); Li et al

    Additionally, following the evaluation methodology of prior Video-LLM works Lin et al. (2023); Li et al. (2024a;b); Ahn et al. (2024), we report the LLMscore in Table

    2023

  10. [27]

    alongside the aligned models (SFT and DPO). 21 Published as a conference paper at ICLR 2025 Figure 17: Additional examples of model prediction from the unaligned model, the model aligned using Supervised Fine-Tuning (SFT), and the model aligned using Direct Preference Optimiza...

    2025

  11. [28]

    If the question cannot be answered using the video content, state that it is unanswerable and provide a reason

    and VLM-RLAIF (Ahn et al., 2024)). Importantly, this performance gap is not attributable to data imbalance, as the UVQA dataset was carefully balanced across these three categories during training, as detailed in Section 5.1.This finding highlights that models generally strugg...

    2024

  12. [29]

    5All annotators have TOEFL iBT scores above 100 and hold at least a bachelor’s degree

    model. 5All annotators have TOEFL iBT scores above 100 and hold at least a bachelor’s degree. 25 Published as a conference paper at ICLR 2025 Table 6:Out-of-Distribution Evaluation of Answerability on Alignment and Absolute Perfor- mance. Base Modelf(·) Alignment Performance A...

    2025

  13. [2008]

    Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A

    doi: 10.1109/TSMCC.2008.919172. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3668–3678,

    2008

  14. [2015]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing

    doi: 10.1109/CVPR.2015.7298990. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decod- ing. InProceedings of the IEEE/CVF Conference on Compu...

    2015

  15. [2016]

    Alignment for honesty

    Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. arXiv preprint arXiv:2312.07000,

    arXiv

  16. [2018]

    Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson

    URL https: //arxiv.org/abs/1802.05451. Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. Incorporating structured representations into pretrained vision & language models using scene graphs. In Houda Bouamor, Juan ...

    2023 arXiv

  17. [2019]

    Video-LLaMA: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Yansong Feng and Els Lefever (eds.),Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp....

    2023

  18. [2020]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al

    URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal...

    2020 arXiv

  19. [2022]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi

    URL https: //arxiv.org/abs/2201.12086. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a. URL https:// arxiv.org/abs/2301.12597. Kunchang Li, Yinan He, Yi Wang, Y...

    2025 arXiv

  20. [2023]

    doi: 10.18653/v1/2023

    Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.870. URLhttps://aclanthology.org/2023.emnlp-main.870. Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing

    2023 doi

  21. [2024]

    URL https: //aclanthology.org/2024.acl-long.52

    Association for Computational Linguistics. URL https: //aclanthology.org/2024.acl-long.52. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report.arXiv...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.