Pith. sign in

REVIEW 4 major objections 8 minor 48 references

Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large multimodal models that answer video questions often rely on prior knowledge and textual context rather than actual video order when temporal inconsistencies are introduced, and a preference-optimization method called PanoDPO…

desk verdict The benchmark evaluation of temporal robustness is real and worth refereeing, but the PanoDPO mitigation section has a load-bearing sign error in Eq. (5) that makes the reported gains uninterpretable as written. read the letter →

arxiv 2505.14405 v1 pith:473LPK4R submitted 2025-05-20 cs.CV

Jiafeng Liang , Shixin Jiang , Xuan Dong , Ning Wang , Zheng Chu , Hui Su , Jinlan Fu , Ming Liu
show 2 more authors
See-Kiong Ng Bing Qin
This is my paper
classification cs.CV
keywords temporalrobustnessvideolargemultimodalmodelsshortcutlearninginconsistencydirectpreferenceoptimizationbenchmarkquestionansweringprior-knowledgebias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current video-capable large multimodal models (LMMs) do not robustly perceive temporal order: when a video's event sequence is made inconsistent with common sense or with the accompanying text, most models fall back on prior knowledge or on textual context instead of the video. To make this measurable, the paper builds TemRobBench, 1,686 multiple-choice QA pairs from 562 instructional videos, with visual perturbations (shuffled event clips at light or severe disorder) and textual perturbations (shuffled event descriptions at absolute or relative disorder), each paired with a 'shortcut' option that matches the pre-perturbation order. Across 16 open and closed LMMs, over 59 percent of wrong answers under perturbation are shortcuts, and option-shuffling shows that much of the remaining 'correct' accuracy is guesswork. The paper then introduces PanoDPO, a direct preference optimization variant that jointly supervises video, question, and response preferences, and reports that it raises true accuracy and lowers flip rates on two 7B backbones while preserving general video QA performance.

What carries the argument

The machine that carries the argument is a two-part construction. TemRobBench is a benchmark built from 562 COIN instructional videos and 1,686 multiple-choice QA pairs; each question is posed in a Clean setting and an Adversarial setting where either the video clips are shuffled (light or severe disorder) or the event descriptions in the prompt are shuffled (absolute or relative disorder). Every adversarial question includes one correct option, one shortcut option matching the pre-perturbation order, and two distractor options, so a model's answer reveals whether it used video order, prior knowledge, or textual context. The evaluation adds Flip Rate (how many clean-correct answers switch to the shortcut), Weak Flip Rate (how many switch to any wrong answer), and true accuracy via four option-shuffling rounds. PanoDPO is a preference-optimization objective that adds video-conditioned and question-conditioned DPO terms to the standard response-level DPO term, trained on rejected videos produced by shuffling, cropping, or blanking frames and rejected questions produced by GPT-4o-generated misleading context.

What would settle it

Rotate the correct option through all four positions and replace the shortcut option with a semantically neutral distractor in the same TemRobBench questions; if models no longer preferentially pick the original shortcut, the measured flip is partly position bias rather than prior-knowledge or text reliance. Separately, evaluating PanoDPO on held-out perturbation families would settle whether the gain is generalized robustness or adaptation to training artifacts.

Watch

Extended reading notes

Core claim

The central discovery is that current LMMs exhibit two temporally specific shortcut behaviors under inconsistency. In the intrinsic case, tampering with the video's event order makes models answer from scripted common-sense orderings; in the extrinsic case, tampering with the event descriptions in the prompt makes models answer from the text. On TemRobBench, every model's accuracy drops in the adversarial setting, with leading models such as Qwen2-VL and InternVL-2.5 losing more than half their clean accuracy and GPT-4o dropping from 67.1 to 16.8 percent on intrinsic temporal shortcuts. Flip rates reach 83.7 percent for some models, meaning large majorities of previously correct answers switch to the curated shortcut. When correct option positions are shuffled over four rounds, true accuracy is dramatically lower than raw accuracy (e.g., under 25 percent for several 7B models), indicating guessing rather than perception. The paper's proposed PanoDPO, which optimizes response, video, and question preferences jointly, shifts the model's log-likelihood difference toward correct answers and roughly doubles true accuracy on LLaVA-OV-7B (8.8 to 16.6 for intrinsic; 11.7 to 15.3 for extrinsic) while reducing flip rates.

Load-bearing premise

The diagnosis assumes that when a model selects the curated shortcut option under perturbation, the cause is over-reliance on prior knowledge or textual context rather than option-position bias or generic confusion, and that training on the same perturbation families used in the benchmark measures generalized robustness rather than adaptation to those artifacts.

Editorial extensions

If this is right

  • Clean-set accuracy is not a trustworthy measure of temporal understanding; option-shuffling or consistency checks are needed to separate perception from guessing.
  • Frame-level corruption benchmarks that add noise or blur are insufficient because temporal-order perturbations expose a failure mode they miss.
  • Instruction prompting to 'watch the video and ignore prior knowledge' does not fix the shortcut, indicating the failure is in learned preferences rather than instruction following.
  • Optimizing video and question preferences jointly, not just response preferences, is a viable route to temporal robustness and does not degrade general video QA performance.
  • Visual temporal perturbations are more misleading than textual perturbations for most models, suggesting that the textual modality already receives stronger supervision during training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stress test for generalization: PanoDPO should be evaluated on perturbation families it never saw during training (e.g., reversed event durations, inserted unrelated events, or reordered sub-actions within an event); if its gains shrink there, it is adapting to artifacts rather than restoring temporal perception.
  • The intrinsic/extrinsic shortcut taxonomy could be connected to hallucination diagnostics: the same mechanism may explain why video LMMs hallucinate event order in open-ended QA, where no multiple-choice shortcut option exists.
  • The likelihood-difference measure (log p(correct) minus log p(shortcut)) could be repurposed as a confidence or calibration signal for video QA systems in deployment, flagging inputs where the model is at risk of shortcutting.
  • If the benchmark is extended to longer, multi-task videos, the T-Acc protocol could serve as a cheap 'perception audit' before trusting a model's video answers in real applications.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper introduces TemRobBench, a video temporal-robustness benchmark built from 562 COIN videos and 1,686 multiple-choice QA pairs. It applies two perturbation families: visual shuffling of event clips ('intrinsic temporal shortcut') and shuffling of event descriptions in the prompt ('extrinsic temporal shortcut'), and evaluates 16 LMMs under clean and adversarial settings using accuracy, flip rate (FR), weak flip rate (WFR), and a 3-out-of-4 option-shuffling 'true accuracy' (T-Acc). The evaluations show large accuracy drops and high flip rates, which the authors interpret as over-reliance on prior knowledge and textual context. To mitigate the problem, the authors propose PanoDPO, which combines standard DPO with video-conditioned (DPOv) and text-conditioned (DPOt) preference objectives, and report T-Acc/FR improvements on two 7B models plus preservation of general capability on VideoMME, LongVideoBench, and ActivityNet-QA.

Significance. The diagnosis that current video LMMs are fragile to simple temporal inconsistencies is supported by a broad 16-model evaluation with coherent accuracy drops and high flip rates; the option-shuffling control is a useful attempt to separate guessing from stable perception, and using COIN as a non-finetuning source reduces leakage concerns. The PanoDPO idea—jointly supervising video, question, and response preferences—is interesting, and the general-capability checks are appropriate. However, the mitigation claims are not yet established: Eq. (5) appears to reverse the chosen/rejected preference, the training perturbation families overlap with the evaluation perturbations, and the results are single-run with no released code or benchmark. These issues need to be resolved before the paper's central claims can be accepted.

major comments (4)
  1. [§4.2, Eq. (5)] The DPOt objective reverses the chosen and rejected conditions. In the DPOm and DPOv equations in the same section, the first log-ratio inside the sigmoid is the preferred condition and the second is the dispreferred condition. Since Section 4.3 constructs rejected questions by adding the perturbation c to the original question xw, the intended preference is (xw, yw) over (xw+c, yw); as written, Eq. (5) increases the model likelihood on the perturbed question and decreases it on the original question, i.e., the opposite of the stated preference. This is not a cosmetic typo: if the code implements Eq. (5) literally, the reported PanoDPO gains in Table 3 cannot be attributed to the panoramic objective described in the paper; if the code implements the correct direction, the equation must be fixed. Because no code or checkpoints are provided, the reader cannot determine which case holds.
  2. [§4.3 vs. §2.2] The PanoDPO evaluation is at least partly circular. The rejected videos in PanoDPO are constructed by shuffling frames, cropping frames, and replacing frames with blank regions, and the rejected questions are generated by GPT-4o to be 'plausible but misleading' (Section 4.3, Appendix B). The TemRobBench adversarial examples are built from the same broad families—clip shuffling for intrinsic shortcuts and shuffling/perturbing event descriptions for extrinsic shortcuts. The observed T-Acc/FR improvements may therefore reflect adaptation to these specific artifact types rather than a general increase in temporal robustness. The authors should demonstrate generalization on disjoint perturbation families (e.g., different shuffle granularities, different textual corruption types, or perturbations not used in training) and report those results.
  3. [§5.2, Tables 3 and 4] The mitigation results are single runs with no variance estimates, confidence intervals, or significance tests, and the improvements are modest (e.g., LLaVA-OV T-Acc ITS 9.6→16.6; LLaVA-Hound ITS 0.9→8.9). In addition, the T-Acc metric uses an arbitrary 3-out-of-4 threshold without comparing against a chance-level or text-only baseline, so the claim that T-Acc isolates 'genuine' temporal perception is not fully supported. The benchmark and evaluation code are not released, which is especially problematic for a benchmark paper: the community cannot verify the numbers or reproduce the exact multiple-choice parsing. Please report multiple seeds with standard deviations, provide a control baseline for T-Acc, and release the code and benchmark.
  4. [§4.3 and Appendix B] The preference data are generated automatically, with rejected videos produced by algorithmic edits and rejected questions produced by GPT-4o using the prompt in Fig. 8. No human validation, inter-annotator agreement, or even qualitative examples of the generated perturbation text are provided. Since PanoDPO's entire effect depends on the preference signal being correct, the paper should include a quality analysis of the preference pairs (e.g., human-annotated correctness of chosen/rejected labels, or automatic checks) and show sample perturbed questions.
minor comments (8)
  1. [§5.3] The cross-references are swapped: the ablation discussion refers to 'Tab. 5' but should refer to Table 4, and the general-capability discussion refers to 'Tab. 4' but should refer to Table 5.
  2. [§4.2] In the paragraph after Eq. (5), 'where vw is the chosen question' should be 'where xw is the chosen question'; the text is describing the question-conditioned term.
  3. [§3.3] The sentence 'some LMMs (e.g., VILA1.5 (Xu et al., 2024))' cites the wrong reference; VILA1.5 is from Lin et al. (2024b), while Xu et al. (2024) is PLLaVA.
  4. [§3.3] 'rely on gambly guess' is a typo; it should be 'rely on a random guess' or 'gamble'.
  5. [§2.1] The heading 'Benchmark Design Principal' should be 'Benchmark Design Principle'.
  6. [Fig. 2 and §1] The figure caption uses 'Internal' and 'External' while the text uses 'Intrinsic' and 'Extrinsic'; the terminology should be unified.
  7. [Fig. 4 and Appendix C] The benchmark statistics report 6.4 events per video, while Appendix C states that COIN videos have 3.91 step segments on average; please clarify whether the figure reports the selected subset rather than the full COIN distribution.
  8. [§3.1, Eq. (1)] The scoring rule 'if yi in ˆyi' is not formally defined for multiple-choice outputs; please specify how responses are parsed (e.g., letter-only matches, substring matches, no-answer cases).

Circularity Check

1 steps flagged · score 6.0 of 10

PanoDPO's text-conditioned objective reverses the declared preference direction; as written, it directly optimizes the exact adversarial-text condition used in the benchmark, making part of the reported robustness gain reduce by construction.

  1. self definitional [Section 4.2 (L_DPOt equation) and Section 4.3 (Preference Data Construction)]
    "Similar to the DPO v, the text-conditioned DPO t includes tuples pairs (vw, xw, yw) and (vw, (xw + c), yw) with the question as the only variable, and its optimization objective can be formulated as: LDPOt =−logσ( βlog πθ(yw|xw+c,vw)/πref(yw|xw+c,vw) −βlog πθ(yw|xw,vw)/πref(yw|xw,vw) ) ... Furthermore, to acquire rejected questions, we introduce perturbations into the original questions."

    In DPO the first log-ratio is the chosen (preferred) condition and the second is the rejected condition. Inserting xw+c in the positive term and xw in the negative term makes the perturbed question the chosen condition, but Section 4.3 declares perturbed questions to be rejected. Minimizing this loss raises πθ(yw | xw+c, vw) and lowers πθ(yw | xw, vw). The Adversarial ETS setting of TemRobBench measures exactly whether the model answers yw under a perturbed question, so the reported ETS T-Acc/FR gains are the literal training objective rather than an emergent robustness property. If the implementation flips the terms, Eq. (5) is a sign typo; either way, the manuscript as written does not support the claimed panoramic preference direction.

full rationale

TemRobBench itself is not circular: it is built from COIN raw annotations with explicit visual/text perturbations and four-option QA, and the evaluation of 16 LMMs is an external measurement. The shortcut-attribution assumption (that choosing the curated option indicates prior-knowledge/textual reliance) is a validity assumption, not a circular reduction. The main circularity risk is in PanoDPO's text condition: the DPOt equation places the perturbed question xw+c in the positive (chosen) log-ratio and the original question xw in the negative (rejected) term, while Section 4.3 defines perturbed questions as rejected. If implemented literally, the model is trained to increase the correct answer's likelihood under exactly the kind of perturbed questions used in the benchmark's Adversarial ETS setting, so the reported ETS gains reduce to the training objective rather than to the stated preference. If the code flips the terms, Eq. (5) is a sign typo and the manuscript does not let the reader verify the method. The external general benchmarks (VideoMME, LongVideoBench, ActivityNet-QA) independently show that PanoDPO does not degrade broad capability, and the DPOm and DPOv terms are self-consistent, so the circularity is partial rather than total. The overlap between PanoDPO's rejected-data perturbations (shuffle/crop/replace, GPT-4o text perturbation) and TemRobBench's perturbation families is a further limitation on any claim of generalized robustness, but it is not by itself a definitional circularity because the evaluation data is held out and the paper provides independent external benchmarks.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the correctness of COIN annotations, exact-match answer scoring, the option-shuffle reliability protocol, and the validity of GPT-4o-generated perturbation text. Hyperparameters beta, crop ratio, and replacement ratio are chosen by hand without sensitivity analysis. No new physical or architectural entities are postulated; the introduced items are named behavioral categories, a benchmark, and a training objective.

free parameters (4)
  • DPO temperature beta = 0.1
    Set by hand in Section 5.1; controls the strength of all three preference losses and is not swept or justified.
  • Rejected-video crop ratio = 20% of frame area
    Chosen for crop corruption in Appendix B; no ablation across ratios.
  • Blank-frame replacement ratio = 50% of frames
    Chosen for replace corruption in Appendix B; no ablation across ratios.
  • True-accuracy voting threshold = 3 of 4 rounds
    T-Acc definition in Section 3.3; the threshold is adopted from prior work but sets the reliability bar.
assumptions (5)
  • domain assumption COIN annotations correctly encode the canonical event order used as ground truth and as the intrinsic shortcut.
    Section 2.2 assumes the unedited clip order is the common-sense prior and that the edited order is the correct video-derived answer.
  • domain assumption Exact string matching of the generated answer to one of the multiple-choice options is a valid measure of model choice.
    Section 3.1 defines Acc via 'Score_i = 1 if y_i in yhat_i', which ignores partial answers and answer-format variability.
  • standard math Bradley-Terry preference model underlies the DPO/PanoDPO objectives.
    Section 4.1 derives the optimization from the Bradley-Terry model, a standard assumption for DPO.
  • domain assumption GPT-4o-generated perturbation text is a suitable proxy for real misleading textual context.
    Appendix B uses a prompt instructing GPT-4o to mislead the model; no human validation or filtering of generated samples is reported.
  • domain assumption Option shuffling with three-of-four agreement removes guessing.
    Section 3.3 uses this threshold to define T-Acc; independence of the four shuffled rounds is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency." pith.science (2026). https://pith.science/paper/473LPK4R

@misc{pith2026250514405,
  author       = {Pith},
  title        = {Pith review of: Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/473LPK4R}},
  note         = {Machine review of arXiv:2505.14405}
}
read the original abstract

Large Multimodal Models (LMMs) have recently demonstrated impressive performance on general video comprehension benchmarks. Nevertheless, for broader applications, the robustness of their temporal analysis capability needs to be thoroughly investigated yet predominantly ignored. Motivated by this, we propose a novel temporal robustness benchmark (TemRobBench), which introduces temporal inconsistency perturbations separately at the visual and textual modalities to assess the robustness of models. We evaluate 16 mainstream LMMs and find that they exhibit over-reliance on prior knowledge and textual context in adversarial environments, while ignoring the actual temporal dynamics in the video. To mitigate this issue, we design panoramic direct preference optimization (PanoDPO), which encourages LMMs to incorporate both visual and linguistic feature preferences simultaneously. Experimental results show that PanoDPO can effectively enhance the model's robustness and reliability in temporal analysis.

Figures

Figures reproduced from arXiv: 2505.14405 by the authors.

Figure 1
Figure 1. An example of the Intrinsic Temporal Shortcut [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Response distribution when asking the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the TEMROBBENCH. The benchmark emphasizes evaluating the model’s robustness against temporal inconsistency, especially take intrinsic shortcuts (over-reliance on prior knowledge) and extrinsic shortcuts (over-reliance on textual context). We construct inconsistencies with knowledge and textual context by shuffling video clips and event descriptions, and design corresponding shortcut answers to verify the… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comprehensive statistics from different per [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Accuracy (left) and flip rate (right) of two [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overview of the PanoDPO. Vanilla DPO (a) expects LMMs to learn response preferences only. PanoDPO [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The discriminative ability of the backbone [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The prompt for generating textual-conditioned preference data. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 15 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  4. [4]

    Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. 2025. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. In ICLR

    work page 2025

  5. [5]

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. 2024. https://doi.org/10.48550/ARXIV.2406.04325 Sharegpt4video: Improving video understanding and generation with better captions . CoRR, abs/2406.04325

    arXiv doi:10.48550/arxiv.2406.04325 2024

  6. [6]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. https://doi.org/10.48550/ARXIV.2406.07476 Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms . CoRR, abs/2406.07476

    arXiv doi:10.48550/arxiv.2406.07476 2024

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  8. [8]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://openreview.net/forum?id=YicbFdNTTy An image is worth 16x16 words: Transformers for image recognition at scale . In 9th International C...

    work page 2021

Show all 48 references
  1. [9]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. https://doi.org/10.48550/AR...

    2024 doi arXiv

  2. [10]

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. https://doi.org/10.1109/CVPR.2015.7298698 Activitynet: A large-scale video benchmark for human activity understanding . In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 201...

    2015

  3. [11]

    Bo Hu, Meng Zhang, Chenfei Xie, Yuanhe Tian, Yan Song, and Zhendong Mao. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.970 RESEMO: A benchmark chinese dataset for studying responsive emotion from social media content . In Findings of the Association for Computational Lin...

    2024 doi

  4. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, ...

    2023 doi arXiv

  5. [13]

    Kang - il Lee, Minbeom Kim, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2024. https://doi.org/10.48550/ARXIV.2406.08702 Vlind-bench: Measuring language priors in large vision-language models . CoRR, abs/2406.08702

    2024 doi arXiv

  6. [14]

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024 a . http://papers.nips.cc/paper\_files/paper/2024/hash/1e69ff56d0ebff0752ff29caaddc25dd-Abstract-Datasets\_and\_Benchmarks\_...

    2024

  7. [15]

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024 b . http://papers.nips.cc/paper\_files/paper/2024/hash/1e69ff56d0ebff0752ff29caaddc25dd-Abstract-Datasets\_and\_Benchmarks\_...

    2024

  8. [16]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 c . https://doi.org/10.48550/ARXIV.2408.03326 Llava-onevision: Easy visual task transfer . CoRR, abs/2408.03326

    2024 doi arXiv

  9. [17]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 d . https://doi.org/10.48550/ARXIV.2407.07895 Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models . CoRR, abs/2407.07895

    2024 doi arXiv

  10. [18]

    Jinmin Li, Kuofeng Gao, Yang Bai, Jingyun Zhang, Shutao Xia, and Yisen Wang. 2024 e . https://doi.org/10.48550/ARXIV.2403.13507 Fmm-attack: A flow-based multi-modal adversarial attack on video-based llms . CoRR, abs/2403.13507

    2024 doi arXiv

  11. [21]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024 a . https://aclanthology.org/2024.emnlp-main.342 Video-llava: Learning united visual representation by alignment before projection . In Proceedings of the 2024 Conference on Empirical Methods in Natu...

    2024

  12. [23]

    Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. 2024 a . https://doi.org/10.48550/ARXIV.2412.14167 Videodpo: Omni-preference alignment for video diffusion generation . CoRR, abs/2412.14167

    2024 doi arXiv

  13. [24]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024 b . https://doi.org/10.18653/V1/2024.FINDINGS-ACL.517 Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics, A...

    2024 doi

  14. [25]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  15. [26]

    OpenGVLab. 2024. https://internvl.github.io/blog/2024-12-05-InternVL-2.5/ Internvl2.5: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    2024

  16. [27]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  17. [28]

    Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. 2024. https://arxiv.org/abs/2212.08044 Benchmarking robustness of multimodal image-text models under distribution shift . Preprint, arXiv:2212.08044

    2024 arXiv

  18. [29]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct preference optimization: Your language model is sec...

    2023

  19. [30]

    Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. 2025. Videoworld: Exploring knowledge learning from unlabeled videos. arXiv preprint arXiv:2501.09781

    2025 arXiv

  20. [31]

    Rawat, and Vibhav Vineet

    Madeline Schiappa, Shruti Vyas, Hamid Palangi, Yogesh S. Rawat, and Vibhav Vineet. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/de6ff07cbd222c10d694c2b2f732aceb-Abstract-Datasets\_and\_Benchmarks.html Robustness analysis of video-language models against visual and ...

    2022

  21. [32]

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. https://doi.org/10.1109/CVPR.2019.00130 COIN: A large-scale dataset for comprehensive instructional video analysis . In IEEE Conference on Computer Vision and Pattern Reco...

    2019

  22. [33]

    Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen

    Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024 a . https://aclanthology.org/2024.emnlp-main.460 mdpo: Conditional preference optimization for multimodal large language models . In Proceedings of the 2024 Conference on Empirical ...

    2024

  23. [34]

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. 2023. https://doi.org/10.1109/CVPR52729.2023.01398 Videomae V2: scaling video masked autoencoders with dual masking . In IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2023

  24. [35]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 b . https://doi.org/10.48550/ARXIV.2409.1...

    2024 doi arXiv

  25. [36]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. 2024 c . https://doi.org/10.1007/978-3-031-73013-9\_23 Intern...

    2024 doi

  26. [37]

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024 a . http://papers.nips.cc/paper\_files/paper/2024/hash/329ad516cf7a6ac306f29882e9c77558-Abstract-Datasets\_and\_Benchmarks\_Track.html Longvideobench: A benchmark for long-context interleaved video-language understanding . I...

    2024

  27. [38]

    Shujin Wu, Yi Fung, Sha Li, Yixin Wan, Kai - Wei Chang, and Heng Ji. 2024 b . https://aclanthology.org/2024.findings-emnlp.454 MACAROON: training vision-language models to be your engaged partners . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miam...

    2024

  28. [39]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat - Seng Chua. 2021. https://doi.org/10.1109/CVPR46437.2021.00965 Next-qa: Next phase of question-answering to explaining temporal actions . In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-2...

    2021

  29. [40]

    Yuxi Xie, Guanzhen Li, Xiao Xu, and Min - Yen Kan. 2024. https://aclanthology.org/2024.findings-emnlp.775 V-DPO: mitigating hallucination in large vision language models via vision-guided direct preference optimization . In Findings of the Association for Computational Linguis...

    2024

  30. [41]

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See - Kiong Ng, and Jiashi Feng. 2024. https://doi.org/10.48550/ARXIV.2404.16994 Pllava : Parameter-free llava extension from images to videos for video dense captioning . CoRR, abs/2404.16994

    2024 doi arXiv

  31. [42]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 doi arXiv

  32. [43]

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. https://doi.org/10.48550/ARXIV.2408.04840 mplug-owl3: Towards long image-sequence understanding in multi-modal large language models . CoRR, abs/2408.04840

    2024 doi arXiv

  33. [44]

    Chenyu Yi, Siyuan Yang, Haoliang Li, Yap - Peng Tan, and Alex C. Kot. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/2a38a4a9316c49e5a833517c45d31070-Abstract-round2.html Benchmarking the robustness of spatial-temporal models against corruptions . In ...

    2021

  34. [45]

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. https://doi.org/10.1609/AAAI.V33I01.33019127 Activitynet-qa: A dataset for understanding complex web videos via question answering . In The Thirty-Third AAAI Conference on Artificial Intelli...

    2019 doi

  35. [46]

    Runhao Zeng, Xiaoyong Chen, Jiaming Liang, Huisi Wu, Guangzhong Cao, and Yong Guo. 2024. https://doi.org/10.1109/CVPR52733.2024.01729 Benchmarking the robustness of temporal action detection models against temporal corruptions . In IEEE/CVF Conference on Computer Vision and Pa...

    2024

  36. [47]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. https://doi.org/10.1109/ICCV51070.2023.01100 Sigmoid loss for language image pre-training . In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 11...

    2023

  37. [48]

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. 2024. https://doi.org/10.48550/ARXIV.2404.01258 Direct preference optimization of video large multimodal models from language ...

    2024 doi arXiv

  38. [49]

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai - Man Cheung, and Min Lin. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a97b58c4f7551053b0512f92244b0810-Abstract-Conference.html On evaluating adversarial robustness of large vision-language models ...

    2023

  39. [50]

    Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, and Jing Jiang. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.683 ROME: evaluating pre-trained vision-language models on reasoning beyond visual common sense . In Findings of the Association for Computation...

    2023 doi

  40. [51]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Caiwan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2024. https://openreview.net/forum?id=QmZKc7UZCy Languagebind: Extending video-language pretraining to n-modality...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.