Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniEval: a bilingual benchmark for models that must fuse vision, audio, and text — and current models score under 65 percent.

desk verdict A useful bilingual omni-modal benchmark, but the central 'full-modal collaboration' claim is undercut by the paper's own ablations and missing modality-necessity statistics. read the letter →

arxiv 2506.20960 v2 pith:2HXMEEKZ submitted 2025-06-26 cs.CV cs.AI

Yiman Zhang , Ziheng Luo , Qiangyu Yan , Wei He , Borui Jiang , Xinghao Chen , Kai Han This is my paper
classification cs.CVcs.AI
keywords omni-modalbenchmarkaudio-visualunderstandingtemporalgroundingbilingualevaluationmultimodallargelanguagemodelsvideoquestionansweringopen-ended
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OmniEval is a new evaluation benchmark with 810 audio-visual videos, 2,617 question–answer pairs, and 12 task types, built to test omni-modal models that must jointly reason over visual, auditory, and textual inputs. The paper constructs tasks explicitly designed to require collaboration among all three modalities, including a fine-grained temporal Grounding task that asks models to locate exact moments or intervals in video. It then evaluates six omni-modal models and reports that even the best tested system (Gemini 2.5 Pro) reaches only 64.56 overall, with Qwen2.5-Omni-7B at 61.02 and VITA-1.5 at 8.14. The larger claim is that OmniEval provides a challenging, valid, bilingual platform for measuring whether a model can construct coherent understanding from all modalities together, not just from one stream. If correct, the benchmark would offer a reusable instrument for guiding future omni-modal model development.

What carries the argument

The central object is the OmniEval dataset itself, built through a multi-stage pipeline: videos are gathered from existing benchmarks and web platforms, captions are obtained or generated with Qwen2.5-VL-70B, speech is transcribed with an ASR system, and then large models generate open-ended questions and convert them into multiple-choice form. Each pair is manually reviewed for clarity, relevance, factual accuracy, and the number of modalities required, then classified into 12 subtask categories, including a dedicated Grounding category with moment-based and time-span-based questions. The evaluation protocol adapts metrics to task type: grounding moment answers are scored against an FPS-dependent temporal threshold, time-span answers use Intersection over Union with a 0.5 threshold, counting answers are extracted by an LLM and compared exactly, and other open-ended answers are scored by LLM similarity.

What would settle it

Take a random sample of OmniEval questions and give human annotators only the video captions and ASR transcripts, with no access to the video or audio. If annotators can answer a large majority of these questions correctly, the central claim that the benchmark requires raw audio and video would be contradicted.

Watch

Extended reading notes

Core claim

The paper claims that OmniEval is the first benchmark to combine full-modal collaboration, bilingual coverage (Chinese and English), mixed open-ended and multiple-choice formats, and fine-grained event grounding in a single evaluation protocol for omni-modal models. Its central finding is that state-of-the-art omni-modal models perform far below what full-modal understanding would require, and worse, their performance is driven mainly by text captions and subtitles rather than by raw audio or video. In the paper's ablations, adding raw video frames to audio-plus-captions inputs sometimes lowers overall scores (for MiniCPM-O 2.6, from 56.54 down to 27.79), and adding raw audio to video-plus-subtitles inputs also frequently degrades results. The paper presents this as evidence that current omni-modal models are not yet truly integrating visual and auditory signals, and that OmniEval exposes this gap more sharply than benchmarks that rely on static images, single-language content, or caption-only tasks.

Load-bearing premise

The benchmark is only valid as a test of full-modal collaboration if the questions truly cannot be answered from captions and transcripts alone, so a reader should check whether the human judgement step actually ensured this for every question.

Editorial extensions

If this is right

  • If OmniEval is accepted as a reliable instrument, model developers gain a concrete target: improving true audio-visual fusion rather than caption-following, since captions and subtitles are not supplied at inference time.
  • The Grounding subtask provides a quantitative way to track progress on temporal localization in omni-modal models, an ability that existing benchmarks treat only coarsely or not at all.
  • The bilingual split makes it possible to measure language-specific weaknesses in omni-modal understanding, potentially guiding multilingual training data collection.
  • The observed performance gap between MC and OE questions suggests that future model improvements should be validated on open-ended generation, not only on choice-based accuracy.
  • The finding that raw video and audio often add no benefit — and sometimes hurt — implies that model architectures must be redesigned so that non-textual streams contribute rather than introduce noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own ablations hint at a stronger conclusion than the authors state: if captions and subtitles alone nearly match or exceed full multi-modal inputs on many questions, then portions of OmniEval may be answerable without grounding in raw video or audio, so the benchmark's 'full-modal collaboration' premise needs direct verification.
  • A testable extension would be to create an adversarial version of OmniEval where captions are deliberately mismatched with the audio or video content; if models still answer correctly, it would confirm that they rely on superficial text cues.
  • The adaptive grounding threshold, which becomes more lenient when few frames are extracted, could over-credit models that output rough timestamps; a reader could check whether performance on grounding questions changes dramatically when the threshold is tightened.
  • The benchmark could be extended to more languages beyond Chinese and English, and to longer videos, to test whether the current findings generalize across linguistic and temporal scales.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces OmniEval, a bilingual (Chinese and English) benchmark for evaluating omni-modal models that take video, audio, and text as input. The benchmark comprises 810 audio-visual synchronized videos and 2,617 QA pairs, split into open-ended and multiple-choice formats, across 12 task types including a fine-grained temporal grounding task. QA pairs are generated with LLM/MLLM assistance from video captions and ASR transcripts, then manually curated. The authors evaluate six omni-modal models, reporting that Gemini 2.5 Pro achieves the highest overall score (64.56), while open-source models lag substantially, and they present ablations (Tables 6 and 7) showing that textual captions and subtitles contribute most of the performance, with raw video and audio often adding little or even hurting performance. The central claim is that OmniEval measures full-modal collaboration, requiring models to integrate visual, auditory, and textual information.

Significance. If the central claim is validated, OmniEval would be a valuable resource: it is one of few benchmarks for omni-modal models with a substantial number of videos, bilingual coverage, a mix of open-ended and multiple-choice questions, and a fine-grained grounding task. The authors ship the benchmark and code, which is a practical strength. However, the paper's defining contribution—that the tasks require full-modal collaboration—is not supported by the evidence presented. The QA generation pipeline relies on captions and ASR transcripts rather than raw video or audio, and the ablations in Section 4.3 show that these textual proxies dominate model performance. Without explicit evidence that a large fraction of questions are unanswerable from text alone, the benchmark may primarily measure language-grounded reasoning rather than multi-modal integration. The evaluation methodology also depends on an unnamed proprietary LLM judge for open-ended questions without calibration to human judgments. These issues are fixable with additional analysis, so the work has potential, but the current version does not substantiate its headline claim.

major comments (4)
  1. [Sec. 3.3.3 (Manual Curation) and Tables 6–7]
  2. [Sec. 4.1.2 (Evaluation for Counting and other OE Q&A pairs)]
  3. [Sec. 4.3 (Impact of Visual Information and Audio Information)]
  4. [Sec. 4.2 (Results on OmniEval)]
minor comments (4)
  1. [Throughout]
  2. [Eq. (1)]
  3. [Sec. 3.3.1 (Removing overly simple samples)]
  4. [Sec. 3.2 (Data Collection and Preprocessing Pipeline)]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OmniEval is an external benchmark; the reported scores are measurements rather than fitted predictions, and the construction pipeline is transparent.

full rationale

OmniEval is an evaluation instrument, not a fitted model, so none of the reported scores (Tables 4-7) is a prediction that reduces by construction to a fitted parameter. The construction pipeline is transparent: automated QA generation uses Qwen2.5-VL-70B captions and Volcano Engine ASR transcripts (Sec 3.3.1), and manual curation includes annotators watching the videos and checking that questions are answerable from the video (Sec 3.3.3). The removal of samples that all helper models answer correctly is a difficulty filter, not a mechanism that forces any particular model ranking or score. The paper's ablations in Sec 4.3 do suggest that captions and subtitles, rather than raw video or audio, drive performance, and the Judgement step's modality-necessity distribution is not reported; however, this is a construct-validity concern about what the benchmark measures, not circularity, because the paper does not derive its central claim from the very assumption it is trying to support. No load-bearing self-citation chain, imported uniqueness theorem, or definitional equation equating the claimed result to its inputs is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five domain assumptions and three hand-chosen thresholds. The most consequential assumptions are that text-drafted questions genuinely require audio-visual integration, and that the unnamed proprietary LLM judge reproduces human scoring; both are unverified in the paper. The benchmark introduces no invented entities such as particles or forces; the Grounding task is a task label, not an entity, and it maps onto existing temporal-localization practice (LongVALE, DeVE-QA).

free parameters (3)
  • time span IoU threshold tau_time_span = 0.5
    Hand-chosen success threshold for time-span grounding evaluation (Eq. 2). Directly determines which grounding answers count as correct and therefore all Grounding sub-scores; no sensitivity analysis is reported.
  • moment threshold tau_ts = min(1/FPS, video_duration/max_frame)
    Hand-designed tolerance for moment-based grounding (Eq. 1). The formula is reasonable, but the choice of the min-rule is arbitrary and no ablation of the threshold is given.
  • speech subdensity filter threshold = 0.5 word density
    Videos with subdensity below 0.5 are excluded (Sec 3.2). This curates the corpus toward dialogue-heavy content and is a free selection knob that shapes the benchmark's scope and difficulty.
assumptions (5)
  • domain assumption LLM-generated QA pairs from captions and transcripts are valid, answerable benchmark questions
    Entered in Sec 3.3.1; the whole benchmark rests on the belief that questions drafted from text summaries probe genuine multimodal understanding rather than text artifacts.
  • domain assumption Qwen2.5-VL-70B captions and Volcano Engine ASR transcripts faithfully represent the videos
    Entered in Sec 3.2; no error analysis of captions or transcripts is provided, and errors here propagate into every QA pair and into the Sec 4.3 'text helps more than video' conclusion.
  • domain assumption Manual review by five annotators guarantees clarity, accuracy, and modality requirements of the QA pairs
    Entered in Sec 3.3.3; no inter-annotator agreement, annotation guidelines, or quality metrics are reported.
  • domain assumption A proprietary LLM judge scores open-ended answers consistently with human judgment
    Entered in Sec 4.1; the judge is unnamed and uncalibrated, and the 0-1 similarity score has no stated pass threshold.
  • ad hoc to paper Removing questions answerable by all tested models improves benchmark validity without biasing difficulty
    Entered in Sec 3.3.1 Step 6; this filter uses the same model families that are later evaluated, so the 'challenging' property is partly definitional.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs." pith.science (2026). https://pith.science/paper/2HXMEEKZ

@misc{pith2026250620960,
  author       = {Pith},
  title        = {Pith review of: OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HXMEEKZ}},
  note         = {Machine review of arXiv:2506.20960}
}
read the original abstract

In this paper, we introduce OmniEval, a benchmark for evaluating omni-modality models like MiniCPM-O 2.6, which encompasses visual, auditory, and textual inputs. Compared with existing benchmarks, our OmniEval has several distinctive features: (i) Full-modal collaboration: We design evaluation tasks that highlight the strong coupling between audio and video, requiring models to effectively leverage the collaborative perception of all modalities; (ii) Diversity of videos: OmniEval includes 810 audio-visual synchronized videos, 285 Chinese videos and 525 English videos; (iii) Diversity and granularity of tasks: OmniEval contains 2617 question-answer pairs, comprising 1412 open-ended questions and 1205 multiple-choice questions. These questions are divided into 3 major task types and 12 sub-task types to achieve comprehensive evaluation. Among them, we introduce a more granular video localization task named Grounding. Then we conduct experiments on OmniEval with several omni-modality models. We hope that our OmniEval can provide a platform for evaluating the ability to construct and understand coherence from the context of all modalities. Codes and data could be found at https://omnieval-benchmark.github.io/.

Figures

Figures reproduced from arXiv: 2506.20960 by the authors.

Figure 1
Figure 1. An grounding example in OmniEval. OmniEval requires integrating both visual and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The left diagram depicts task question quantity grouped by functional categories in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

    cs.AI 2026-03 conditional novelty 6.5 of 10

    SocialOmni jointly evaluates speaker ID, turn-entry timing, and interruption phrasing on 12 OLMs and finds perception accuracy decouples from socially appropriate generation.

  2. MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    MultiRef-Compass is a 350-sample benchmark and 14-metric protocol for multi-reference-to-audio-video generation; current models still fail at reference binding and audio-visual consistency.

Reference graph

Works this paper leans on

60 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716–23736, 2022

    2022

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015

    work page 2015

  3. [3]

    From pre-training to fine-tuning: An in-depth analysis of large language models in the biomedical domain

    Agnese Bonfigli, Luca Bacco, Mario Merone, and Felice Dell’Orletta. From pre-training to fine-tuning: An in-depth analysis of large language models in the biomedical domain. Artificial Intelligence in Medicine, 157:103003, 2024

    work page 2024

  4. [4]

    Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline, 2017

    Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline, 2017

    2017

  5. [5]

    Cg-bench: Clue-grounded question answering benchmark for long video understanding

    Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg-bench: Clue-grounded question answering benchmark for long video understanding. arXiv preprint arXiv:2412.12075, 2024

    arXiv 2024

  6. [6]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021

    arXiv 2021

  7. [7]

    Emotionlines: An emotion corpus of multi-party conversations

    Shao-Yen Chen, Chung-Chi Hsu, Chien-Chung Kuo, and Lun-Wei Ku. Emotionlines: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379, 2018

    arXiv 2018

  8. [8]

    VideoLLaMA 2: Advanc- ing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. VideoLLaMA 2: Advanc- ing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. arXiv preprint arXiv:2406.07476, 2024

    arXiv 2024

Show all 60 references
  1. [9]

    Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  2. [10]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    Fleurs: Few-shot learning evaluation of universal representations of speech, 2022

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech, 2022

    2022

  4. [12]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098–89124, 2024

    2024

  5. [13]

    Finevideo

    Miquel Farre, Andi Marafioti, Lewis Tunstall, Leandro V on Werra, and Thomas Wolf. Finevideo. https://huggingface.co/datasets/HuggingFaceFV/finevideo, 2024

    2024

  6. [14]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  7. [15]

    VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction. arXiv preprint a...

    2025 arXiv

  8. [16]

    Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. arXiv preprint arXiv:2411.19772, 2024

    2024 arXiv

  9. [17]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google and DeepMind. Gemini 2.5: Our most intelligent ai model, 2025

    2025

  10. [18]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  11. [19]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. VizWiz Grand Challenge: Answering Visual Questions from Blind People. arXiv preprint arXiv:1802.08218, 2018

    2018 arXiv

  12. [20]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300, 2021

    2009 arXiv

  13. [21]

    TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation, page 198–208

    François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation, page 198–208. Springer International Publishing, 2018

    2018

  14. [22]

    Worldsense: Evaluating real-world omnimodal understanding for multimodal llms, 2025

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms, 2025

    2025

  15. [23]

    Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    2025 arXiv

  16. [24]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017

    2017

  17. [25]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referitgame: Re- ferring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787–798. Association for Computati...

    2014

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In The Tenth International Conference on Learning Representations (ICLR), 2023

    2023

  19. [27]

    Baichuan-omni-1.5 technical report

    Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report. arXiv preprint arXiv:2501.15368, 2025

    2025

  20. [28]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  21. [29]

    Omnibench: Towards the future of universal omni-language models

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272, 2024

    2024

  22. [30]

    Omnibench: Towards the future of universal omni-language models, 2025

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. O...

    2025

  23. [31]

    Streamingbench: Assessing the gap for mllms to achieve streaming video understanding

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:2411.03628, 2024

    2024 arXiv

  24. [32]

    Clotho-aqa: A crowdsourced dataset for audio question answering, 2022

    Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering, 2022. 11

    2022

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  26. [34]

    MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281, 2024

    2024 arXiv

  27. [35]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. arXiv preprint arXiv:2310.02255, 2024

    arXiv

  28. [36]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Findings of the Association for Computationa...

    2022

  29. [37]

    Spoken question answering and speech continuation using spectrogram-powered LLM

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered LLM. In The Twelfth International...

    2024

  30. [38]

    V oxceleb: a large-scale speaker identification dataset

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. V oxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017

    2017 arXiv

  31. [39]

    Minicpm-o 2.6: A gpt-4o-level mllm for vision, speech, and multi- modal live streaming on your phone

    OpenBMB Team. Minicpm-o 2.6: A gpt-4o-level mllm for vision, speech, and multi- modal live streaming on your phone. Online; OpenBMB Notion Page, 2025. Available at: https://github.com/OpenBMB/MiniCPM-o

    2025

  32. [40]

    Librispeech: An asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015

    2015

  33. [41]

    Plummer, Liwei Wang, Christopher M

    Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. IJCV, 123(1):74–93, 2017

    2017

  34. [42]

    MELD: A multimodal multi-party dataset for emotion recognition in conversation

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gaurav Naik, Rada Mihalcea, and Erik Cambria. MELD: A multimodal multi-party dataset for emotion recognition in conversation. In Proceedings of the AAAI Conference on Artificial Intelligence: Workshop on Explainable AI (xAI...

    2019

  35. [43]

    Question-answering dense video events

    Hangyu Qin, Junbin Xiao, and Angela Yao. Question-answering dense video events. arXiv preprint arXiv:2409.04388, 2024

    2024 arXiv

  36. [44]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  37. [45]

    Towards VQA Models That Can Read.arXiv preprint arXiv:1904.08920, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read.arXiv preprint arXiv:1904.08920, 2019

    1904 arXiv

  38. [46]

    Deepseek v3: Scaling large language models with sparse mixture-of- experts

    DeepSeek AI Team. Deepseek v3: Scaling large language models with sparse mixture-of- experts. Technical Report arXiv:2505.14283, DeepSeek Inc., 2025

    2025 arXiv

  39. [47]

    Language models are few-shot learners

    Open AI Team. Language models are few-shot learners. Technical Report arXiv:2005.14165, arXiv preprint arXiv:2005.14165, jul 2020. Preprint

    2005 arXiv

  40. [48]

    Gpt-4 technical report

    Open AI Team. Gpt-4 technical report. Technical Report arXiv:2303.08774, OpenAI, 2024

    2024 arXiv

  41. [49]

    Qwen2.5: Technical Report

    Qwen AI Team. Qwen2.5: Technical Report. Technical Report arXiv:2412.15115, Alibaba Cloud, January 2025. Preprint. 12

    2025 arXiv

  42. [50]

    CogVLM: Visual Expert for Pretrained Language Models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. CogVLM: Visual Expert for Pretrained Language Models. arXiv preprint arXiv:2311.03079, 2024

    2024 arXiv

  43. [51]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weim- ing Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A More Robust and Challenging Multi-Task Language Unde...

    2024 arXiv

  44. [52]

    Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks

    Haiyang Xu, Qinghao Ye, Xuan Wu, Ming Yan, Yuan Miao, Jiabo Ye, Guohai Xu, Anwen Hu, Yaya Shi, Guangwei Xu, et al. Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks. arXiv preprint arXiv:2306.04362, 2023

    2023 arXiv

  45. [53]

    Qwen2.5-Omni Technical Report

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni Technical Report. arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  46. [54]

    Air-bench: Benchmarking large audio-language models via generative comprehension, 2024

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. Air-bench: Benchmarking large audio-language models via generative comprehension, 2024

    2024

  47. [55]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.TACL, 2:67–78, 2014

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.TACL, 2:67–78, 2014

    2014

  48. [56]

    Berg, and Yuandong Tian

    Licheng Yu, Mark Barrow, Tamara L. Berg, and Yuandong Tian. Modeling context in referring expressions. In Proceedings of the 14th European Conference on Computer Vision (ECCV), volume 9908 of Lecture Notes in Computer Science, pages 3–19. Springer, 2016

    2016

  49. [57]

    HellaSwag: Can a Machine Really Finish Your Sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  50. [58]

    Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition, 2022

    Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition, 2022

    2022

  51. [59]

    Zhang and M

    H. Zhang and M. O. Shafiq. Survey of transformers and towards ensemble learning using transformers for natural language processing. Journal of Big Data, 11:25, 2024

    2024

  52. [60]

    Yin and Yang: Balancing and answering binary visual questions

    Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 13

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.