Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces a hand-built, expert-validated bilingual benchmark and claims it shows current multimodal AI models vary sharply across industrial domains.

desk verdict MME-Industry fills a real gap as a cross-industry bilingual benchmark, but its data-integrity claim rests on a statistically invalid no-image test and an unreleased dataset. read the letter →

arxiv 2501.16688 v1 pith:7B2GE65L submitted 2025-01-28 cs.CL

Dongyi Yi , Guibo Zhu , Chenglin Ding , Zongshu Li , Dong Yi , Jinqiao Wang This is my paper · ORCID
classification cs.CL
keywords multimodallargelanguagemodelsMLLMevaluationindustrialbenchmarkvisualquestionansweringbilingualdataleakagedomainexpertisemultiple-choiceQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to fill a gap in AI evaluation: most vision-language benchmarks test everyday photos, documents, or charts, not real industrial work. It introduces MME-Industry, a benchmark of 1,050 manually written and expert-validated multiple-choice questions, 50 per industrial sector across 21 fields, each paired with a high-resolution image and offered in Chinese and English. The design deliberately removes OCR-style questions and adds items that need specialized sector knowledge, so a model's score is meant to reflect visual understanding plus domain reasoning. If the benchmark is sound, the reported ranking—led by Qwen2-VL-72B-Instruct at 78.66% Chinese and 75.04% English—gives a practical comparison of how well current multimodal models handle industrial visual tasks.

What carries the argument

The central mechanism is the benchmark itself: 1,050 image-question pairs, fifty per sector across 21 industrial domains, each with a high-resolution image (average 1110×859 pixels), one correct answer, three similar wrong answers, and a fifth 'no corresponding features' option, in parallel Chinese and English versions. The manual, expert-validated construction is what is supposed to guarantee data integrity and prevent leakage from public datasets. The extra 'E' option also does functional work: it gives models a channel to refuse when they cannot see image features, and it turns API failures, oversized images, or illegal inputs into a measurable rejection instead of forcing a guess.

What would settle it

A decisive check would be a contamination probe: take the same 1,050 questions, withhold the images, and force a choice among only the four content options; if any model scores at or above the 25% level, the claimed absence of data leakage is false. A second probe would swap in different questions for the same images: if scores collapse, the benchmark is measuring image-question coupling rather than robust industrial reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that MME-Industry captures genuine industrial multimodal ability rather than text recognition or memorized answers. The evidence is the contrast between image-present and image-absent conditions: with images, the ten tested models ranged up to 78.66% (CN) and 75.04% (EN), while without images the average was 16.46% (CN) and 16.78% (EN), values the authors read as below random guessing and therefore as proof that the questions were not answerable from language alone. The benchmark's construction is the load-bearing part: experts hand-picked 50 images per sector, wrote one question with four plausible wrong options plus a fifth 'no corresponding features' rejection option, validated the content for professional accuracy, and translated the whole set into English. The authors also report that technical sectors such as Electronics, Chemicals, and Light Industry are relatively strong for the best models, while Finance, Education, and Building Materials are hard across the board. On the paper's own terms, that pattern is what a useful industrial evaluation should reveal: where current models can be deployed and where they still need domain-specific work.

Load-bearing premise

The entire data-integrity claim depends on one test: models given questions without images answered below 25%, which the paper treats as random chance, even though the real random baseline with five options is 20% and the models may simply be picking the 'no matching features' option.

Editorial extensions

If this is right

  • If the benchmark is sound, the reported ranking is a practical shortlist: Qwen2-VL-72B-Instruct and Claude-3.5-Sonnet lead on industrial visual QA, while smaller open models like MiniCPM-V-2.6 trail sharply.
  • The consistently low performance on Finance, Education, and Building Materials suggests those sectors need more specialized training data before deployment, not just larger general-purpose models.
  • The large image-versus-no-image gaps show that the benchmark is testing visual understanding, so future model gains likely require better high-resolution image encoding rather than better language reasoning alone.
  • The bilingual Chinese/English versions expose language-specific weaknesses, such as MiniCPM-V-2.6 scoring 18.47% in Chinese versus 29.04% in English, which points to language coverage as an independent failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger leakage check than the paper's text-only test would hide the image and also remove the 'E' escape option; that would separate memorized question-text answers from genuine guessing and address the fact that with five options random chance is 20%, not the 25% the paper uses.
  • The current scoring rewards instruction-following because the full model response is used without extracting the letter; a future version could report both raw-answer accuracy and extraction-based accuracy to show how much the rankings depend on obedience to the 'answer with one letter' prompt.
  • Read per sector, the table could support deployment decisions, such as which models are safe to use in chemical or electronics inspection, and this per-domain calibration is a direct use the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MME-Industry, a bilingual (Chinese/English) multiple-choice benchmark of 1,050 image-question pairs across 21 industrial domains, with 50 questions per domain, manually crafted and expert-validated. Ten multimodal large language models are evaluated on the benchmark in both languages, with Qwen2-VL-72B-Instruct reported as the best performer (78.66% CN, 75.04% EN). To support the benchmark's data-integrity claim, the authors run a text-only 'no image' experiment and report average accuracies of 16.46% (CN) and 16.78% (EN), which they interpret as evidence against data leakage.

Significance. If the benchmark is released and its integrity verified, MME-Industry would fill a genuine gap in the evaluation of MLLMs for specialized industrial domains, and the bilingual design plus the deliberate removal of OCR-only questions are valuable choices. The paper currently provides no public dataset or code, however, and the no-leakage evidence is statistically flawed and internally contradicted by the reported per-model results. With corrected analysis, a valid contamination check, and public release, the benchmark could be a useful community resource.

major comments (3)
  1. [§4.2, Table 3] The no-image leakage test uses an incorrect random baseline and the aggregated average is uninformative. With five options (A–E), chance accuracy is 20%, not the 25% stated in the text. The pooled CN average of 16.46% is below 20%, but it is dragged down by models that essentially always choose E when no image is present (e.g., Gemini-1.5-Flash at 0%). The test is further undermined by per-model results: JT-VL-Chat-V3.0 achieves 30.76% CN and 32.76% EN without image input, and GLM-4V-Flash achieves 32.28% CN and 29.04% EN—both above 20% and even above the paper's own 25% threshold. Per-domain parenthetical scores in Tables 4 and 5 include values such as 58% for JT-VL-Chat-V3.0 in the Medical CN domain, showing that some models answer correctly from the text alone on substantial subsets. The conclusion that 'the model is not relying on data leakage' is therefore not supported by this experiment; a valid contamination check requires either direct membership testing or a controlled study with held-out distractors.
  2. [§3.1, §5] The benchmark, which is the paper's central contribution, is not made available. No repository, URL, or supplementary data is provided, and the conclusion only lists 'establishing open-source platforms' as future work. Without public release, the community cannot reproduce the reported scores, verify the manual/expert construction, check for image or text contamination, or use the benchmark for evaluation. A benchmark paper should include a release plan, ideally with an anonymous repository link in the submitted version, and a statement of dataset access terms.
  3. [§3.1, §4] The paper reports no human performance, inter-annotator agreement, or other quantitative validation of the ground-truth answers. The claim that questions are 'validated by domain experts' is not substantiated with numbers, and no evidence is given that the images actually contain the information needed to answer the questions without external knowledge or memorization. A human baseline would calibrate the difficulty of the benchmark and provide a reference point for interpreting the MLLM scores; without it, the benchmark's validity as a ground-truth evaluation set is not fully established.
minor comments (6)
  1. [Abstract, §3.1] The abstract contains a grammatical error ('21 distinct domain' should be '21 distinct domains'), and Section 3.1 has typos such as 'recoginze' and 'rasies' that should be corrected.
  2. [§4.1] The model-selection paragraph says InternVL2.5-78B-MPO was selected, but the list of evaluated models in the same section reads 'InternVL2.5-78B' without '-MPO', while Table 2 uses the '-MPO' suffix; please clarify which model was actually evaluated.
  3. [Table 1, Figure 1] Option E is worded inconsistently between Table 1 ('There are no corresponding features in the image') and Figure 1 ('No corresponding features in the picture'); the options should be standardized across the paper.
  4. [§3.2] The 21 task categories are listed in a single run-on paragraph with entries like 'Textile:various' and 'Steel:identification'; adding paragraph breaks or a table would greatly improve readability.
  5. [§4.2] The no-image experiment does not report the fraction of responses that were option E; reporting this fraction would help distinguish abstention from genuine text-based answering and would make the aggregate averages interpretable.
  6. [Figure 2] Figure 2 shows only 6 of the 10 evaluated models, and the caption's '6 top MLLMs' is not justified by any stated criterion; please explain the subset selection or show all models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MME-Industry is an independently constructed benchmark and all reported results are measurements, not derivations from the benchmark's construction assumptions.

full rationale

The paper derives no quantitative claim from its own construction choices. MME-Industry is presented as a manually created evaluation set (1,050 expert-validated QA pairs over 21 domains), and the model scores in Tables 2-5 are obtained by running external MLLMs on that fixed set; there are no fitted parameters, no equations whose output is defined to match an input, and no 'prediction' that is statistically forced by a fit. The data-integrity discussion in Section 4.2 uses a no-image control, but the conclusion that models are not relying on leakage is an empirical inference, not a definitional equivalence: even if the control is statistically weak (the random baseline with five options including an abstention option is 20%, not 25%, and per-model no-image scores in Tables 4-5 exceed 25%), the weakness is a validity concern about a supporting test, not a circular step that makes the benchmark score equal its input. The paper's citations to prior benchmarks (MME, MMBench, MMStar, MMMU, etc.) are contextual and comparative; none is a load-bearing self-citation, and the authors' own prior work is not invoked to force a conclusion. Renaming is not present: the benchmark is a new dataset, not a re-expression of known results under new coordinates. Therefore no enumerated circularity pattern applies; the honest finding is no significant circularity (score 0), with the data-integrity control flagged as a separate correctness risk.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's claims rest primarily on the reliability of expert annotations and the validity of the leakage check, neither of which is independently verified in the paper. No new theoretical entities or fitted parameters are introduced.

assumptions (4)
  • domain assumption Domain expert annotation and validation ensure that all questions are correct, unambiguous, and industrially relevant.
    The paper relies on self-reported expert review (Section 3.1) without providing annotator credentials, inter-annotator agreement, or a human baseline.
  • domain assumption Text-only performance below the random-choice threshold indicates no data leakage.
    The paper's leakage test (Section 4.2) assumes that low accuracy without images proves the model has not memorized the QA pairs. This is weak because models may recognize images without being able to answer from text alone, and the stated 25% threshold is incorrect for 5 options.
  • domain assumption The 21 selected industrial sectors are representative of diverse industrial applications.
    The paper asserts comprehensive coverage (Section 3.2) but does not justify why these 21 sectors were chosen or how they generalize to the full space of industrial tasks.
  • domain assumption Standard evaluation settings (temperature 0.2, top-p 0.95, max 64 tokens) yield reliable and comparable model scores.
    These settings are chosen in Section 4.1 without sensitivity analysis or repeated runs, so score stability is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark." pith.science (2026). https://pith.science/paper/7B2GE65L

@misc{pith2026250116688,
  author       = {Pith},
  title        = {Pith review of: MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7B2GE65L}},
  note         = {Machine review of arXiv:2501.16688}
}
read the original abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), numerous evaluation benchmarks have emerged. However, comprehensive assessments of their performance across diverse industrial applications remain limited. In this paper, we introduce MME-Industry, a novel benchmark designed specifically for evaluating MLLMs in industrial settings.The benchmark encompasses 21 distinct domain, comprising 1050 question-answer pairs with 50 questions per domain. To ensure data integrity and prevent potential leakage from public datasets, all question-answer pairs were manually crafted and validated by domain experts. Besides, the benchmark's complexity is effectively enhanced by incorporating non-OCR questions that can be answered directly, along with tasks requiring specialized domain knowledge. Moreover, we provide both Chinese and English versions of the benchmark, enabling comparative analysis of MLLMs' capabilities across these languages. Our findings contribute valuable insights into MLLMs' practical industrial applications and illuminate promising directions for future model optimization research.

Figures

Figures reproduced from arXiv: 2501.16688 by the authors.

Figure 1
Figure 1. MME-Industry: Our benchmark covers over 20 industries. The diagram shows a subset of the MME Industry Evaluation Set, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of 6 top MLLMs across 21 industries in English evaluate. The maximum score for each industry is 100. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 9,356-pair Chinese multimodal financial benchmark reveals that state-of-the-art multimodal LLMs, including GPT-4V, still score below 53% on objective and 39% on subjective financial chart tasks.

Reference graph

Works this paper leans on

36 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv 2023

  2. [3]

    Qwen technical report

    [Bai et al., 2023a] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    arXiv

  3. [4]

    Qwen-vl: A frontier large vision- language model with versatile abilities

    [Bai et al., 2023b] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

  4. [5]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    [Chen et al., 2024] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv 2024

  5. [8]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    [Dong et al., 2024] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420,

    arXiv 2024

  6. [9]

    Palm-e: An embodied multimodal language model

    [Driess et al., 2023] Danny Driess, Fei Xia, Mehdi SM Saj- jadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378,

    arXiv 2023

  7. [10]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models,

    [Duan et al., 2024] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models,

    work page 2024

  8. [11]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    [Fu et al., 2023] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xi- awu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Ron- grong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394,

    arXiv 2023

Show all 36 references
  1. [12]

    Chatglm: A fam- ily of large language models from glm-130b to glm-4 all tools

    [GLM et al., 2024] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A fam- ily of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793,

    2024 arXiv

  2. [13]

    Lora: Low-rank adaptation of large language models

    [Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    2021 arXiv

  3. [15]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

    [Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning , pages 19730–19742. PMLR,

    2023

  4. [16]

    Monkey: Image resolution and text label are important things for large multi-modal models

    [Li et al., 2024] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision ...

    2024

  5. [17]

    Mmbench: Is your multi-modal model an all- around player? arXiv preprint arXiv:2307.06281,

    [Liu et al., 2023a] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all- around player? arXiv preprint arXiv:2307.06281,

    arXiv

  6. [18]

    On the hidden mystery of ocr in large multi- modal models

    [Liu et al., 2023b] Yuliang Liu, Zhang Li, Biao Yang, Chun- yuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multi- modal models. arXiv preprint arXiv:2305.07895,

    arXiv

  7. [19]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models

    [Liu et al., 2024a] Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935,

    arXiv

  8. [20]

    Chartqa: A bench- mark for question answering about charts with visual and logical reasoning

    [Masry et al., 2022] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    2022 arXiv

  9. [22]

    Training language models to follow instruc- tions with human feedback

    [Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in neural informa- tion p...

    2022

  10. [23]

    Multilayer perceptron (mlp)

    [Taud and Mas, 2018] Hind Taud and Jean-Franccois Mas. Multilayer perceptron (mlp). Geomatic approaches for modeling land change scenarios , pages 451–455,

    2018

  11. [25]

    Internlm: A multilingual language model with progressively enhanced capabilities,

    [Team, 2023] InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities,

    2023

  12. [26]

    Llama 2: Open foundation and fine-tuned chat models

    [Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    2023 arXiv

  13. [27]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning

    [Wang et al., 2023a] Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574,

    arXiv

  14. [28]

    Cogvlm: Visual expert for pretrained language models

    [Wang et al., 2023b] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079,

    arXiv

  15. [29]

    Baichuan 2: Open large- scale language models

    [Yang et al., 2023] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large- scale language models. arXiv preprint arXiv:2309.10305,

    2023 arXiv

  16. [30]

    mplug-owl: Modulariza- tion empowers large language models with multimodality

    [Ye et al., 2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304.14178,

    2023 arXiv

  17. [31]

    mplug-owl3: Towards long image-sequence under- standing in multi-modal large language models,

    [Ye et al., 2024] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence under- standing in multi-modal large language models,

    2024

  18. [32]

    Mm-vet: Evaluating large multi- modal models for integrated capabilities

    [Yu et al., 2023] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308.02490,

    2023 arXiv

  19. [33]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    [Yue et al., 2024] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/...

    2024

  20. [34]

    Mm-llms: Recent advances in multimodal large language models

    [Zhang et al., 2024a] Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601,

    arXiv

  21. [35]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257,

    [Zhang et al., 2024b] Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios ...

    arXiv

  22. [36]

    Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models

    [Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  23. [2018]

    Gemini: a family of highly capable multimodal models

    [Team et al., 2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Mil- lican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    2023 arXiv

  24. [2021]

    Scaling up visual and vision-language representation learning with noisy text su- pervision

    [Jia et al., 2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text su- pervision. In International conference on machine learn...

    2021

  25. [2022]

    Docvqa: A dataset for vqa on document images

    [Mathew et al., 2021] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2200–2209,

    2021

  26. [2023]

    Openflamingo: An open-source framework for training large autoregressive vision-language models

    [Awadalla et al., 2023] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv pr...

    2023 arXiv

  27. [2024]

    Sharegpt4v: Improving large multi-modal models with better captions

    [Chen et al., 2025] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Com- puter Vision, pages 370–387. Springer,

    2025

  28. [2025]

    Palm: Scaling lan- guage modeling with pathways

    [Chowdhery et al., 2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling lan- guage modeling with pathways. Journal of Machine Learning Research...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.