Pith. sign in

REVIEW 4 major objections 6 minor 57 references

Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VideoLLaMA-2 leads traffic VideoQA models with 57 percent average accuracy.

desk verdict Plausible ranking, but the judge is also a contestant and the reported numbers don't fully cohere, so the headline result isn't reliable yet. read the letter →

arxiv 2412.01132 v1 pith:GB7DSNMM submitted 2024-12-02 cs.CV

Joseph Raj Vishal , Divesh Basina , Aarya Choudhary , Bharatesh Chakravarthi This is my paper · ORCID
classification cs.CV
keywords VideoQuestionAnsweringVision-LanguageModelsLargeLanguageNaturalUnderstandingSpatial-TemporalReasoningMulti-ObjectDetectionTrafficMonitoringSceneAnalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether today's video question answering models can monitor traffic footage, and it tests five leading models on three short clips: two recorded at real-world intersections and one generated by a driving simulator. The central claim is that VideoLLaMA-2 is the strongest of the five, averaging 57 percent accuracy across questions about object detection, temporal events, and complex scene reasoning. The study also argues that, despite this leader, all models fail at multi-object tracking, temporal coherence, and compositional scene understanding, which blocks near-term real-time traffic applications. If correct, the result identifies both the current best architecture for traffic VideoQA and the specific capabilities that must improve next.

What carries the argument

The argument runs on the evaluation protocol rather than on any single model architecture. Two building blocks carry the work: first, a language-model-as-judge design in which GPT-4o assigns a binary semantic-correctness score to each model response by comparing it with a human-written ground truth while deliberately not seeing the video; second, the consistency precision ($cP$) metric, which divides the number of categories where a model answered every question correctly by the total number of categories where it answered either all or none correctly. Questions were generated by GPT-4o from human-filtered templates, organized into easy, moderate, and complex levels, with negated versions of several questions included to probe hallucination.

What would settle it

Have several human annotators independently score the same 54 model responses against the given ground truths, then compare the human rankings with GPT-4o's rankings; if agreement is low or a different model finishes first, the paper's central result is contradicted.

Watch

Extended reading notes

Core claim

On the study's 54-question, three-video evaluation, VideoLLaMA-2 scored 57 percent average accuracy, ahead of GPT-4o (53 percent), Gemini 1.5 Pro (51 percent), InternVL (46 percent), and LLaVA-NeXT-Video-7B (40 percent). The result is produced by a protocol in which GPT-4o, without seeing the footage, marks each model answer as semantically correct or wrong against a human-written ground truth, and a consistency precision ($cP$) metric checks whether a model that answers one question in a category correctly also answers the rest of that category correctly. VideoLLaMA-2 stood out in compositional and negated-question reasoning, while every model showed errors in counting moving objects, tracking objects across time, and interpreting complex scenes. The paper concludes that the current generation of VideoQA models is not yet dependable for real-time traffic monitoring.

Load-bearing premise

All accuracy numbers depend on GPT-4o correctly and impartially judging whether a model's answer carries the same meaning as the human-written ground truth, and no check against human raters or against the video is reported.

Editorial extensions

If this is right

  • VideoLLaMA-2 is the best-performing model of the five evaluated for traffic-focused VideoQA, averaging 57 percent accuracy.
  • At current accuracy levels, none of the five models is dependable for real-time traffic monitoring; all share failure modes in multi-object tracking, temporal coherence, and complex scene interpretation.
  • The open-source evaluation framework combining GPT-4o semantic scoring with consistency precision can rank models on non-benchmark traffic footage without requiring a manually built test dataset.
  • The largest accuracy gains for traffic VideoQA are likely to come from improving object tracking and temporal alignment rather than from scaling language ability alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because GPT-4o both authored the questions and judged the answers, the accuracy ranking may partly reflect a preference for GPT-4o's own answer style; validating the judge against human raters is a direct next step.
  • With only three video clips and 54 questions, the headline 57 percent figure is a first estimate rather than a stable benchmark; repeating the protocol on more clips could reorder the models.
  • The same judge-based protocol could extend to other real-time embodied domains such as warehouse safety or drone surveillance, but the judge-bias concern would need resolving first.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper evaluates five video question answering (VideoQA) models—GPT-4o, LLaVA-NeXT-Video-7B-hf, Gemini 1.5 Pro, Intern-VL, and VideoLLaMA-2—on a self-constructed set of 54 questions over three traffic video sequences (two real-world, one synthetic). The authors use GPT-4o as a semantic-equivalence judge to score model responses against human-authored ground truths, and report category-wise and overall accuracies. They claim VideoLLaMA-2 achieves the highest average accuracy (57%) and stands out in compositional reasoning and consistency, while all models show limitations in multi-object tracking and temporal reasoning. The code and question sets are open-sourced.

Significance. If validated, the study would provide a useful pilot benchmark for traffic-domain VideoQA and a reusable open-source evaluation framework, with the concrete finding that current VideoQA models are not yet reliable for nuanced real-time traffic queries. The strengths are the use of non-benchmark real-world and synthetic traffic sequences, human-authored ground truths, and the public release of the evaluation materials and code. However, the headline ranking is currently fragile: it rests on a two-question margin scored by an unvalidated judge that is itself one of the contestants, with no error bars, no consistency metric reported, and no reconciliation of per-category and overall percentages. The claimed 'state-of-the-art assessment' therefore is not yet supported at the precision claimed.

major comments (4)
  1. [5.2, 4.4, 3.2] All accuracy numbers are produced by GPT-4o acting as a binary semantic-equivalence judge, yet GPT-4o is also one of the five evaluated models (Section 4.4) and the generator of the question set (Section 3.2). The paper labels GPT-4o an 'impartial evaluator' (Section 5.2) but reports no calibration of its judgments against human raters, no alternate judge, and no blinding of model identity in the prompt (Section 5.2.1), which presents responses in a model-labeled table. Because every numerical result in Tables 3-6 depends on this judge, the headline ranking is not verifiable at the claimed precision; please add a human-annotated agreement study (e.g., Cohen's kappa on a sample), use at least one independent judge, and mask model identities in the judge prompt.
  2. [Table 6, Section 6.4] The overall accuracy difference between VideoLLaMA-2 (57%) and GPT-4o (53%) corresponds to roughly 31 versus 29 correct answers out of 54 questions, a margin of two questions. The paper reports no error bars, no confidence intervals, and no repeated runs, despite stating in Section 6.2 that models 'frequently provi[ded] different outputs upon repeated iterations.' A single stochastic run cannot support the claim that VideoLLaMA-2 'stood out' or 'excelled' (Abstract, Section 7). Please provide per-model raw counts, multiple runs or a variance estimate, and a statistical test (or at least a clearly stated binomial interval) for the ranking.
  3. [5.2.1, 7] The consistency precision (cP) metric is defined in Section 5.2.1 and is central to the abstract's claim that VideoLLaMA-2 showed 'answer consistency across related queries' and 'stood out' in consistency, but no cP values are reported anywhere in the paper. Without the cP table (or the per-category consistency data needed to compute it), the consistency component of the central claim is unsupported. Please include the cP results for all models and question categories, or revise the claim accordingly.
  4. [Tables 3-5 and Table 6] The per-category percentages in Tables 3-5 do not transparently combine to the overall averages in Table 6. For instance, if each category contains 6 questions (as implied by Section 3.2), the simple average of VideoLLaMA-2's nine category percentages is about 59%, not 57%; the mismatch may stem from rounding or from unequal category denominators, but neither is stated. Please report the denominator and raw correct counts per category, sequence, and model, and show explicitly how Table 6 is computed from Tables 3-5.
minor comments (6)
  1. [Section 2.1] 'VidedvzdoQA' is a typo for 'VideoQA'.
  2. [Section 6.4] The two paragraphs starting 'Across all sequences...' are duplicated verbatim; remove the duplicate.
  3. [Tables 3-5 and 7-9] Use consistent model names across tables and text (e.g., 'ChatGPT-4o' vs 'GPT-4o', 'LLaV A-NeXT-Video-7B-hf' vs 'LLaVA-NeXT-Video-7B-hf').
  4. [Table 2] VideoLLaMA-2 is marked '✗' for 'Long Video Support', which conflicts with Section 4.2's description of the STC connector as reducing token overload for video sequences; clarify the criterion or the mark.
  5. [Appendix A.1] The prompt in Appendix A.1 asks for '10 questions for each category (30 questions in total)', but Section 3.2 says 18 questions per video with 6 per category; reconcile the counts and state the exact per-category totals.
  6. [Section 3.2] Please state the total number of questions explicitly in the main text (54 follows from 18 per video × 3 videos), to aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported ranking is an empirical evaluation outcome, not a construction forced by the paper's definitions or citations.

full rationale

Walking the paper's derivation chain: there is no formal derivation to reduce. The central claim (VideoLLaMA-2 achieves 57% average accuracy and stands out in consistency) is an empirical ranking produced by a stated evaluation protocol: human-authored ground truth, model responses, and GPT-4o binary semantic-equivalence judgments (Sections 5.2, 5.2.1). The ground-truth answers are not generated by the evaluated models; Appendix A.1 explicitly instructs GPT-4o 'Do not generate the answers' during question curation. The evaluation frameworks cited by the paper, Marcu et al. [5] and Liao et al. [6], are from external research groups, not from the present authors, so no load-bearing self-citation chain supports the conclusion. The most serious concern is that GPT-4o serves simultaneously as question generator (Section 3.2), evaluated contestant (Section 4.4), and sole evaluator (Section 5.2), with no human calibration or alternate judge reported, and Table 6 never presents the cP values defined in Section 5.2.1 despite the consistency claim. That is a real validity and evidence-quality threat, but it is not circularity in the derivation-chain sense: the 57% figure is not equivalent by construction to any input, no equation or definition forces the ranking, and the outcome does not favor the evaluator model (GPT-4o scored 53%, behind VideoLLaMA-2). Therefore no specific circular step can be exhibited from the paper's own equations or citations, and the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The evaluation introduces no new physical or mathematical entities. The only load-bearing choices are the judge model, the hand-labeled difficulty levels, and the representativeness of three selected clips.

free parameters (1)
  • Question difficulty thresholds = Easy, moderate, complex (categorical)
    Each question was hand-categorized into one of three difficulty levels (Section 3.2, Table 1). These labels affect per-category accuracy scores, and no inter-rater reliability or definitional validation is provided.
assumptions (3)
  • domain assumption GPT-4o can judge semantic equivalence between model answers and ground truth without seeing the video
    Section 5.2 states GPT-4o was prompted to score responses as 1 or 0 based on semantic consistency, and this score is the sole basis for all accuracy numbers.
  • domain assumption The three selected video sequences are representative of traffic monitoring tasks
    Section 3.1 justifies selection as "non-benchmark" and containing events of interest, but the sample is three short clips picked by the authors without a sampling scheme.
  • domain assumption Human-authored ground truth answers are correct
    Ground truth answers are used as the reference in GPT-4o scoring (Section 5.2 and Appendix A.2); no independent verification or inter-annotator agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks." pith.science (2026). https://pith.science/paper/GB7DSNMM

@misc{pith2026241201132,
  author       = {Pith},
  title        = {Pith review of: Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GB7DSNMM}},
  note         = {Machine review of arXiv:2412.01132}
}
read the original abstract

Recent advances in video question answering (VideoQA) offer promising applications, especially in traffic monitoring, where efficient video interpretation is critical. Within ITS, answering complex, real-time queries like "How many red cars passed in the last 10 minutes?" or "Was there an incident between 3:00 PM and 3:05 PM?" enhances situational awareness and decision-making. Despite progress in vision-language models, VideoQA remains challenging, especially in dynamic environments involving multiple objects and intricate spatiotemporal relationships. This study evaluates state-of-the-art VideoQA models using non-benchmark synthetic and real-world traffic sequences. The framework leverages GPT-4o to assess accuracy, relevance, and consistency across basic detection, temporal reasoning, and decomposition queries. VideoLLaMA-2 excelled with 57% accuracy, particularly in compositional reasoning and consistent answers. However, all models, including VideoLLaMA-2, faced limitations in multi-object tracking, temporal coherence, and complex scene interpretation, highlighting gaps in current architectures. These findings underscore VideoQA's potential in traffic monitoring but also emphasize the need for improvements in multi-object tracking, temporal reasoning, and compositional capabilities. Enhancing these areas could make VideoQA indispensable for incident detection, traffic flow management, and responsive urban planning. The study's code and framework are open-sourced for further exploration: https://github.com/joe-rabbit/VideoQA_Pilot_Study

Figures

Figures reproduced from arXiv: 2412.01132 by the authors.

Figure 1
Figure 1. An example scenario of a question posed by a traffic analyst and answered by a VideoQA model is based on a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Traffic Video Data Sources - (a) Real-world traffic video captured at a four-way intersection in Tempe, AZ. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Sample Scenarios from traffic Videos - Sequence 1 (top) represents a real-world scenario captured at a [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Shows the architectures of three open-source models: (a) InternVL [ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: (a) Shows an extensive summary of a 45mins lecture provided by GPT-4o, it includes subjection for each topic discussed in the video [7], (b) represents the Gemini Pro VideoQA capabilities the movie Sherlock Jr. of 44mins is passed. The superiority of the model to answe…
Figure 6
Figure 6. Figure 6: An overview of the output tables based on the prompt. (a) are the scores generated on questions taken from [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Average accuracy across question types for video sequence 1 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Average accuracy across question types for video sequence 2 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Average accuracy across question types for video sequence 3 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Average accuracy across all sequences and video types. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 33 canonical work pages

  1. [1]

    Video question answering: Datasets, algorithms and challenges, 2022

    Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answering: Datasets, algorithms and challenges, 2022

    2022

  2. [2]

    Videoqa in the era of llms: An empirical study, 2024

    Junbin Xiao, Nanxin Huang, Hangyu Qin, Dongyang Li, Yicong Li, Fengbin Zhu, Zhulin Tao, Jianxing Yu, Liang Lin, Tat-Seng Chua, and Angela Yao. Videoqa in the era of llms: An empirical study, 2024

    work page 2024

  3. [3]

    Francis, and Alessandro Oltramari

    Ehsan Qasemi, Jonathan M. Francis, and Alessandro Oltramari. Traffic-domain video question answering with automatic captioning, 2023

    work page 2023

  4. [4]

    Carla: An open urban driving simulator, 2017

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator, 2017

    2017

  5. [5]

    Lingoqa: Visual question answering for autonomous driving, 2024

    Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski. Lingoqa: Visual question answering for autonomous driving, 2024

    work page 2024

  6. [6]

    Align and aggregate: Compositional reasoning with video alignment and answer aggregation for video question-answering, 2024

    Zhaohe Liao, Jiangtong Li, Li Niu, and Liqing Zhang. Align and aggregate: Compositional reasoning with video alignment and answer aggregation for video question-answering, 2024

    work page 2024

  7. [7]

    GPT-4o: System Card

    OpenAI. GPT-4o: System Card. Available at: https://cdn.openai.com/gpt-4o-system-card.pdf , 2024. https://cdn.openai.com/gpt-4o-system-card.pdf

    work page 2024

  8. [8]

    Dynamic spatio-temporal graph reasoning for videoqa with self-supervised event recognition

    Jie Nie, Xin Wang, Runze Hou, Guohao Li, Hong Chen, and Wenwu Zhu. Dynamic spatio-temporal graph reasoning for videoqa with self-supervised event recognition. IEEE Transactions on Image Processing, 33:4145– 4158, 2024

    work page 2024

Show all 57 references
  1. [9]

    Recognizing an action using its name: A knowledge-based approach

    Chuang Gan, Yi Yang, Linchao Zhu, Deli Zhao, and Yueting Zhuang. Recognizing an action using its name: A knowledge-based approach. International Journal of Computer Vision, 120:61–77, 2016

    2016

  2. [10]

    Video representation learning with deep neural networks

    Linchao Zhu. Video representation learning with deep neural networks. PhD thesis, 2019

    2019

  3. [11]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017

    2017

  4. [12]

    Vilbert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Informatio...

    2019

  5. [13]

    Compositional attention networks with two-stream fusion for video question answering

    Ting Yu, Jun Yu, Zhou Yu, and Dacheng Tao. Compositional attention networks with two-stream fusion for video question answering. IEEE Transactions on Image Processing, 29:1204–1218, 2020

    2020

  6. [14]

    Visualbert: A simple and performant baseline for vision and language, 2019

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, 2019

    2019

  7. [15]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961–970, 2015

    2015

  8. [16]

    Tgif: A new dataset and benchmark on animated gif description, 2016

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description, 2016

    2016

  9. [17]

    Next-qa:next phase of question-answering to explaining temporal actions, 2021

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question-answering to explaining temporal actions, 2021

    2021

  10. [18]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team et al. Gemini: A family of highly capable multimodal models, 2024

    2024

  11. [19]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024

    2024

  12. [20]

    Ego4d: Around the world in 3,000 hours of egocentric video, 2022

    Kristen Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video, 2022

    2022

  13. [21]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, ...

    2023

  14. [22]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024. 19 SOTA Video Question Answering Models Assessment for Traffic Monitoring Tasks

    2024 arXiv

  15. [23]

    Mist: Medical image segmenta- tion transformer with convolutional attention mixing (cam) decoder

    Md Motiur Rahman, Shiva Shokouhmand, Smriti Bhatt, and Miad Faezipour. Mist: Medical image segmenta- tion transformer with convolutional attention mixing (cam) decoder. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 404–413, J...

    2024

  16. [24]

    A multi-world approach to question answering about real-world scenes based on uncertain input

    Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. ...

    2014

  17. [25]

    G. Awad, K. Curtis, A. A. Butt, J. Fiscus, A. Godil, Y . Lee, A. Delgado, E. Godard, L. Diduch, D. Gupta, D. D. Fushman, Y . Graham, and G. Qu’enot. Trecvid 2023 - a series of evaluation tracks in video understanding. In Proceedings of TRECVID 2023. NIST, USA, 2023

    2023

  18. [26]

    Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events

    Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9878–9888, June 2021

    2021

  19. [27]

    Argos vision: Advanced computer vision solutions, 2024

    Argos Vision. Argos vision: Advanced computer vision solutions, 2024. Accessed: 2024-11-26

    2024

  20. [28]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204–13214, 2024

    2024

  21. [29]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  22. [30]

    Llava-next- interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next- interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  23. [31]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  24. [32]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024

    2024

  25. [33]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024

    2024 arXiv

  26. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu et al. Improved baselines with visual instruction tuning. In NeurIPS, volume 36, 2023

    2023

  27. [35]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu et al. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165, 2021

    2021 arXiv

  28. [36]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv prepri...

    2023 arXiv

  29. [37]

    Internlm2 technical report

    Zheng Cai et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

    2024 arXiv

  30. [38]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  31. [39]

    Minesh Mathew

    et al. Minesh Mathew. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021

    2021

  32. [40]

    Ahmed Masry et. al. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, pages 2263–2279, 2022

    2022

  33. [41]

    Infographicvqa

    Minesh Mathew et al. Infographicvqa. In WACV, pages 1697–1706, 2022

    2022

  34. [42]

    Towards vqa models that can read

    Amanpreet Singh et al. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019

    2019

  35. [43]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023

    2023 arXiv

  36. [44]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A compr...

    2024

  37. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  38. [46]

    Beats: Audio pre-training with acoustic tokenizers

    Chen et al. Beats: Audio pre-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings o...

    2023

  39. [47]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  40. [48]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Chen et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479, 2024

    2024 arXiv

  41. [49]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

    Zhu et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023

    2023 arXiv

  42. [50]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Bain et al. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  43. [51]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Wang et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023

    2023 arXiv

  44. [52]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Changpinyo et al. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558–3568, 2021

    2021

  45. [53]

    A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions

    Urbanek et al. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. arXiv preprint arXiv:2312.08578, 2023

    2023 arXiv

  46. [54]

    Advancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  47. [55]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  48. [56]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  49. [57]

    How many cars can you see?

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, May 2024. 21 SOTA Video Question Answering Models Assessment for Traffic Monitoring Tasks A A...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.