Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reconstruction fidelity is a practical measure of an LMM's image understanding.

desk verdict A useful new evaluation loop, but uncalibrated: the ranking is credible, the absolute scores and pattern-level claims are not yet. read the letter →

arxiv 2411.14062 v2 pith:QCZDWQVC submitted 2024-11-21 cs.CV cs.AIcs.CLcs.LG

Hailang Huang , Yong Wang , Zixuan Huang , Huaqiu Li , Tongwen Huang , Xiangxiang Chu , Richong Zhang This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords automatedevaluationpipelinelargemultimodalmodelstext-to-imagegenerationimageunderstandingdescriptionreconstructionfidelitybenchmarkcosinesimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that an LMM can be said to understand an image only if it can compress that image into a caption from which a text-to-image model can rebuild the same scene. It operationalizes this idea as MMGenBench-Pipeline: the LMM writes a detailed caption, a diffusion model generates a fresh image from it, and an image encoder measures how close the original and regenerated images are. On a newly constructed test set of 1,284 images across 13 visual patterns, the strongest evaluated models reach only moderate reconstruction similarity, with the best scores below 0.6 on a cosine scale, and several models that top existing benchmarks score substantially lower on this test. The paper interprets these results as evidence that fine-grained understanding—counting, orientation, motion, and contextual relations—remains a weak point for current LMMs, and that reconstruction-based evaluation exposes failures that short-answer benchmarks hide.

What carries the argument

The load-bearing mechanism is the reconstruction loop, a three-stage composition: the LMM maps the input image to a caption $P_{\mathrm{gen}} = \mathrm{LMM}(I_{\mathrm{inp}}, \mathrm{prompt})$; a text-to-image model maps that caption to a fresh image $I_{\mathrm{gen}} = G(\epsilon; P_{\mathrm{gen}}, \theta)$; and an image encoder maps both images to embeddings whose cosine similarity defines the SIM-Score. The loop is treated as a lossy compression channel: a caption is good exactly when regenerating from it lands near the original in representation space. Four state-of-the-art text-to-image generators are used to keep the measurement from depending on any one generator, and their rankings are shown to be consistent.

What would settle it

Take a set of human-written captions for the same images (for example, the annotations used to build the test set), feed them through the same four text-to-image models and the same encoder, and compare the resulting similarity scores with the LMMs' scores. If human captions do not score clearly higher, or if the same weak patterns—context, orientation, count, motion—stay weak even with human captions, then the pipeline is measuring generator limitations rather than LMM understanding.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a measurement identity: a caption that preserves the essential information of an image will, after regeneration, produce an image whose representation is close to the original, so reconstruction similarity is a direct readout of the LMM's image understanding. The paper reports that no evaluated model crosses 0.6 on this score; the best open model reaches 0.599 and the strongest closed API model 0.566. Per-pattern scores split cleanly, with coarse attributes like artistic style, surreal content, symbols, and color scoring higher, and fine-grained attributes like contextual relations, orientation, counting, and motion scoring lower. The paper attributes this gap to LMMs' training on short captions and to their tendency to overfit benchmark-style tasks, and it points to short descriptions, instruction-format failures, and safety over-refusal as the main observable failure modes.

Load-bearing premise

The text-to-image models render a correct, detailed caption faithfully enough, and with roughly equal difficulty across all 13 image patterns, so that differences in reconstruction similarity reflect the LMM's description quality rather than the generator's own strengths and weaknesses.

Editorial extensions

If this is right

  • Standard leaderboards can be complemented with a reconstruction-based score that directly penalizes vague or incomplete descriptions.
  • Training LMMs on longer, more detailed image-description pairs should improve their reconstruction scores, since the paper's failure analysis points to short training captions as a bottleneck.
  • Because the pipeline needs only images, it can be pointed at any new domain without manual annotation or task-specific question writing.
  • Per-pattern scores offer a diagnostic checklist—count, orientation, motion, and contextual relations—for comparing models on fine-grained description ability.
  • Instruction-following quality becomes measurable independently of content knowledge through observable formatting failures in generated captions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A calibration run the paper does not report—human-written reference captions through the same generators and encoder—would establish the absolute ceiling; without it, the 0.6-level scores may encode generator difficulty as much as LMM quality.
  • The pattern-level weakness on count, orientation, and motion may be partly inherited from text-to-image models, which are known to struggle with those same attributes, so cross-pattern gaps are best read as a joint system diagnostic.
  • The same loop could be inverted into a training signal: captions that reconstruct well could be up-weighted in instruction tuning, converting the benchmark into a data-curation filter.
  • Because the score depends on the chosen generator and encoder, absolute numbers are only comparable when those components are frozen; upgrading either one silently rescales the leaderboard.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces MMGenBench-Pipeline, a fully automatic evaluation protocol for large multimodal models (LMMs) based on text-to-image reconstruction. For each input image, the LMM under test generates a caption-prompt; one of four text-to-image models (FLUX.1-dev, Stable Diffusion 3.5, Kolors, Lumina) synthesizes an auxiliary image; and the Unicom image encoder is used to compute cosine similarity (SIM-Score) and FID between the original and reconstructed images. The authors construct MMGenBench-Test (1,284 JourneyDB images annotated with 13 image patterns) and MMGenBench-Domain (10,000 images) and evaluate over 50 LMMs. They report that the best models score below 0.600 SIM-Score, that model rankings are consistent across the four generators, and that LMMs are particularly weak on Contextual, Orientation, Count, and Motion patterns, concluding that existing benchmarks overstate basic image understanding and description abilities.

Significance. The pipeline is conceptually attractive as a low-cost, scalable, and domain-agnostic evaluation tool, and the paper has several concrete strengths: the four-generator cross-validation in Fig. 6, the broad model coverage (over 50 LMMs), the planned release of code and data, and the human-verified 13-pattern test set. The central claim—that reconstruction fidelity measures understanding via a compression argument (Sec. 1)—is plausible but is only as strong as the assumption that the text-to-image stage is a faithful and difficulty-uniform channel. The paper does not yet supply the calibration experiment needed to support that assumption, so the absolute and pattern-level conclusions are premature in their current form. With the addition of the calibration and sensitivity analyses described below, the pipeline could become a useful complement to existing benchmarks.

major comments (3)
  1. [Sec. 2.1, Eq. (4), Sec. 4.2] The interpretation of absolute SIM-Score as a measure of LMM understanding is unanchored because no ceiling calibration is performed. To claim that a score below 0.600 indicates that even top LMMs are inadequate describers, one must know what SIM-Score a perfect caption achieves on this dataset. I request an experiment that feeds human-written or ground-truth captions—for example, the original JourneyDB prompts—through the same four generators and the Unicom encoder on the same 1,284 MMGenBench-Test images, reporting overall and per-pattern SIM-Score. If the reconstruction ceiling is near 0.6 for these images, the observed scores reflect generator and encoder limits rather than LMM deficiencies. This experiment is the load-bearing test for the paper's central claim in Sec. 1 and Sec. 4.2.
  2. [Sec. 4.2, Fig. 7, Table 2] The pattern-level conclusion that LMMs are weak on 'Contextual', 'Orientation', 'Count', and 'Motion' is confounded by known weaknesses of current text-to-image models on exactly these dimensions. The four-generator comparison in Fig. 6 shows only that the generators rank LMMs similarly; it does not establish that the generator renders each pattern with roughly equal difficulty. Without a per-pattern ceiling calibration, the observed pattern differences can be fully explained by generator difficulty, and the claim that LMMs are proficient at coarse-grained patterns but weak at fine-grained relationships is unsupported. The per-pattern ceiling experiment from the first major comment should be the basis for any pattern-level claim.
  3. [Eq. (4), Sec. 2.1, Appendix E] The SIM-Score relies on a single image encoder (Unicom) whose sensitivity to the fine-grained attributes the benchmark claims to measure is unvalidated. Even a faithful text-to-image generator would not yield a meaningful score if the encoder cannot distinguish a detailed, accurate caption from a short, vague one that names only coarse content. I request a controlled sensitivity experiment: take a set of captions, systematically degrade specific attributes (e.g., remove object counts, alter spatial relations, change orientation), generate images from each version, and measure whether SIM-Score drops accordingly. This would directly address the concern, acknowledged in Appendix E for table images, that the pipeline may be inaccurate for content that text-to-image and image representation models handle poorly.
minor comments (5)
  1. [Fig. 2(a)] The label 'MCQY/NCaption' appears garbled; it should be split or clarified to identify the different task types shown.
  2. [Throughout] The model name 'LLaV A-OV' contains an internal space in the text and tables; the official name 'LLaVA-OneVision' should be used for consistency.
  3. [Table 3] The row 'XComposer2 [10]' cites reference [10], which is InternVL; the citation should be corrected to the appropriate XComposer reference.
  4. [Fig. 4] The caption should state explicitly that the percentages are relative to the total number of pattern annotations (4,297), not to the 1,284 images, since images can have multiple patterns.
  5. [Sec. 4.3] The paper should state whether the full raw LMM output—including prefixes, suffixes, and safety refusals shown in Fig. 9—was passed to the text-to-image model without parsing, since this affects the interpretation of instruction-following failures and their impact on SIM-Score.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SIM-Score is a directly computed external measurement; the missing ceiling calibration is a validity limitation, not a circular step.

full rationale

The derivation chain is a measurement rather than a derivation: P_gen = LMM(I_inp, P_art) (Eq. 1), I_gen = G(epsilon; P_gen, theta) (Eq. 2), and SIM-Score = cosine(F_Iinp, F_Igen) (Eq. 4). The target claim about LMM understanding is connected to this score only through the motivating 'compression' assumption in Sec. 1 ('A robust understanding of an image implies that LMMs can distill its essential information into text prompts...'), which is an interpretive claim about what the measured quantity means, not a quantity defined in terms of the score or fitted to the score. No parameter is fitted to the evaluated models, no ground-truth caption is used to define the metric, and no external benchmark result is assumed to derive the rankings. GPT-4o is used both to annotate MMGenBench-Test patterns and as an evaluated model, but pattern annotations never enter the SIM-Score computation, so this overlap is not load-bearing. The sole self-citation (ref. [12], MobileVLM v2) appears in a general list of LMM progress in Sec. 1 and does not support any load-bearing step. The reviewer's ceiling-calibration concern (no human-written captions fed through FLUX.1-dev, SD 3.5, Kolors, Lumina to establish what SIM-Score a perfect caption achieves) is a legitimate construct-validity and interpretability limitation of the benchmark, but it does not make the measured score equal to its input by construction. Therefore no circular step is exhibited, and the correct finding is no significant circularity (score 0).

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

No numeric parameters are fitted to data: the generator, encoder, prompts, and metrics are fixed design choices, and the two free parameters above are hand-chosen design elements rather than fitted constants. The load-bearing inputs are domain assumptions about the measurement instrument, chiefly that text-to-image reconstruction fidelity tracks LMM understanding and that rendering difficulty is roughly uniform across patterns; none of these is validated against human-rated descriptions in the paper. No new physical or model entities are introduced: the 13 patterns are a descriptive labeling taxonomy over existing images, not a postulated mechanism.

free parameters (2)
  • 13-pattern taxonomy (MMGenBench-Test) = 13 hand-selected labels, listed in Fig. 1
    Hand-selected from 1,868 GPT-4o-extracted patterns via GPT-4 Turbo summarization and manual filtering (Sec. 3.3). Not fitted to a numeric target, but it defines the pattern-level claims in Fig. 7.
  • caption word-count window in evaluation prompt = 20 to 60 words
    Hand-chosen prompt constraint (Appendix C.1, Fig. 16) that shapes description length and thus what the generator can reconstruct; a design choice, not a fitted constant.
assumptions (6)
  • domain assumption Reconstruction fidelity (image to caption to regenerated image similarity) is a valid proxy for LMM image understanding and description quality.
    The "compression" premise stated in Sec. 1 and formalized as the SIM-Score comparison in Sec. 2.2; it is the core measurement assumption and is not validated against human-rated descriptions or ground-truth captions.
  • domain assumption Near-uniform rendering difficulty of the chosen text-to-image models across the 13 image patterns, so pattern-level SIM differences are attributable to the LMM rather than the generator.
    Implicit in the pattern analysis of Sec. 4.2 / Fig. 7; no ceiling calibration with known-good captions is provided, and T2I models are known to struggle with count, text, and spatial layout content.
  • domain assumption Cosine similarity in Unicom embedding space is an adequate measure of semantic image similarity for this task.
    Unicom is introduced in Sec. 2.1 to 2.2 without validation on this specific task; FID computed on the same features is acknowledged to behave inconsistently across dataset sizes.
  • domain assumption FLUX.1-dev results are representative because the four text-to-image models give consistent SIM and FID orderings in Fig. 6.
    Stated in Sec. 4.1; consistency across generators is argued, but the absolute ceiling remains unmeasured.
  • domain assumption JourneyDB test and validation images are a suitable domain-neutral testbed, and GPT-4o pattern annotations plus human filtering yield correct pattern labels.
    Sec. 3.3; the benchmark inherits JourneyDB's content distribution and uses GPT-4o as annotator with manual checks, so errors in annotation propagate into the pattern stratification.
  • standard math Standard cosine similarity and FID formulas apply as written.
    Sec. 2.2; the formulas are standard, though FID here is computed with Unicom features over 1,284 images, which is outside the usual Inception-based protocol and is sensitive to sample size.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective." pith.science (2026). https://pith.science/paper/QCZDWQVC

@misc{pith2026241114062,
  author       = {Pith},
  title        = {Pith review of: MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCZDWQVC}},
  note         = {Machine review of arXiv:2411.14062}
}
read the original abstract

Large Multimodal Models (LMMs) demonstrate impressive capabilities. However, current benchmarks predominantly focus on image comprehension in specific domains, and these benchmarks are labor-intensive to construct. Moreover, their answers tend to be brief, making it difficult to assess the ability of LMMs to generate detailed descriptions of images. To address these limitations, we propose the MMGenBench-Pipeline, a straightforward and fully automated evaluation pipeline. This involves generating textual descriptions from input images, using these descriptions to create auxiliary images via text-to-image generative models, and then comparing the original and generated images. Furthermore, to ensure the effectiveness of MMGenBench-Pipeline, we design MMGenBench-Test, evaluating LMMs across 13 distinct image patterns, and MMGenBench-Domain, focusing on generative image performance. A thorough evaluation involving over 50 popular LMMs demonstrates the effectiveness and reliability of both the pipeline and benchmark. Our observations indicate that numerous LMMs excelling in existing benchmarks fail to adequately complete the basic tasks related to image understanding and description. This finding highlights the substantial potential for performance improvement in current LMMs and suggests avenues for future model optimization. Concurrently, MMGenBench-Pipeline can efficiently assess the performance of LMMs across diverse domains using only image inputs.

Figures

Figures reproduced from arXiv: 2411.14062 by the authors.

Figure 1
Figure 1. The MMGenBench-Test consists of 13 distinct image patterns, each of which includes several images. The text, accompanied by a corresponding pattern, serves as a concise explanation of that specific image pattern. Please refer to the Appendix B.1 for more details. Abstract Large Multimodal Models (LMMs) demonstrate impressive capabilities. However, current benchmarks predominantly focus on image comprehension in spec… view at source ↗
Figure 2
Figure 2. Comparison between previous benchmarks and MM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An overview of the MMGenBench-pipeline, illustrating the fully automated evaluation process. It starts by receiving user input [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Statistics of MMGenBench-Test, which contains [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: An overview of the MMGenBench-Test benchmark construction process. We first use GPT-4o to extract the image patterns [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The comparative analysis of four different text-to-image [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Model performance by image patterns. Please refer to [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The comparison of SIM-Score with different LMMs on [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Qualitative Results on MMGenBench. We present common problems identified in the experiments, which include issues with [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: A case study of image pattern extraction. GPT-4o is [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: 13 Image Patterns, obtained through GPT-4 Turbo summarization and subsequently verified by humans. Extracted Image Patterns “Surreal”: 2262, “Structural and Physical Characteristics”: 2016, “Position and Contextual Relationship”: 1775, “Geometry”: 1270, “Orientation a…
Figure 12
Figure 12. Figure 12: Images patterns extracted by GPT-4o and ranked in descending order of frequency. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Prompt for Extraction. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Prompt for Summary. 7 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Prompt for Re-annotation. 8 [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Evaluation Pipeline Prompt. Text of Multi-modal Input in Figure [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: More Qualitative Results on MMGenBench. (1/5) 10 [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: More Qualitative Results on MMGenBench. (2/5) 11 [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: More Qualitative Results on MMGenBench. (3/5) 12 [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: More Qualitative Results on MMGenBench. (4/5) 13 [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: More Qualitative Results on MMGenBench. (5/5) 14 [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Evaluating Robustness of Prompt Adherence in Text to Image Models

    cs.CV 2025-07 conditional novelty 4.0 of 10

    New benchmark results show that Stable Diffusion 3.x and Janus Pro models struggle to place simple geometric shapes in the correct image quadrant, with best F1 scores around 0.41 to 0.5.

Reference graph

Works this paper leans on

96 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. 3, 4

    2024

  2. [2]

    Lawrence Zitnick, Dhruv Batra, and Devi Parikh

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016. 2

    2016

  3. [3]

    Pixtral 12b, 2024

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, et al. Pixtral 12b, 2024. 3, 4

    2024

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information pro- cessing systems, 35:23716–23736, 2022. 1

    2022

  5. [5]

    Unicom: Universal and compact representation learning for image re- trieval, 2023

    Xiang An, Jiankang Deng, Kaicheng Yang, Jaiwei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image re- trieval, 2023. 3, 5

    2023

  6. [6]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 1, 5, 6, 7, 3, 4

    arXiv 2023

  7. [7]

    Benchmarking foundation models with language- model-as-an-examiner

    Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xi- aozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language- model-as-an-examiner. NIPS, 36, 2024. 1

    2024

  8. [8]

    Autobench-v: Can large vision-language models benchmark themselves? arXiv preprint arXiv:2410.21259, 2024

    Han Bao, Yue Huang, Yanbo Wang, Jiayi Ye, Xiangqi Wang, Xiuyin Chen, Mohamed Elhoseiny, and Xiangliang Zhang. Autobench-v: Can large vision-language models benchmark themselves? arXiv preprint arXiv:2410.21259, 2024. 1

    arXiv 2024

Show all 96 references
  1. [9]

    Are we on the right way for evaluating large vision-language mod- els? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, et al. Are we on the right way for evaluating large vision-language mod- els? arXiv preprint arXiv:2403.20330, 2024. 1

    2024 arXiv

  2. [10]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 ,

    arXiv

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 3, 4

    2024 arXiv

  4. [12]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. 1

    2024 arXiv

  5. [13]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https:// github.com/open-compass/opencompass , 2023. 2, 3, 4

    2023

  6. [14]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models, 2024. 3, 4

    2024

  7. [15]

    Vlmevalkit: An open- source toolkit for evaluating large multi-modality models,

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open- source toolkit for evaluating large multi-modality models,

  8. [16]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  9. [17]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2

    2023 arXiv

  10. [18]

    Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning, 2024

    Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning, 2024. 2

    2024

  11. [19]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024. 2

    2024

  12. [20]

    Lumina-t2x: Transforming text into any modality, resolution, and dura- tion via flow-based large diffusion transformers, 2024

    Peng Gao, Le Zhuo, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, Chen Lin, Rongjie Huang, Shijie Geng, Renrui Zhang, Junlin Xi, et al. Lumina-t2x: Transforming text into any modality, resolution, and dura- tion via flow-based large diffusion transformers, 2024. 3, 5

    2024

  13. [21]

    Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance. arXiv preprint arXiv:2410.16261, 2024. 3, 4

    arXiv

  14. [22]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024. 3, 4

    2024

  15. [23]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  16. [24]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024

  17. [25]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  18. [26]

    Cogvlm2: Visual language models for image and video understanding, 2024

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding, 2024. 3, 4

    2024

  19. [27]

    Chatgpt for shaping the future of 9 dentistry: the potential of multi-modal large language model

    Hanyao Huang, Ou Zheng, Dongdong Wang, Jiayi Yin, Zi- jin Wang, Shengxuan Ding, Heng Yin, Chuan Xu, Renjie Yang, Qian Zheng, et al. Chatgpt for shaping the future of 9 dentistry: the potential of multi-modal large language model. International Journal of Oral Science, 15(1):29...

    2023

  20. [28]

    Genmac: Compositional text-to-video generation with multi-agent collaboration, 2024

    Kaiyi Huang, Yukun Huang, Xuefei Ning, Zinan Lin, Yu Wang, and Xihui Liu. Genmac: Compositional text-to-video generation with multi-agent collaboration, 2024. 2

    2024

  21. [29]

    Mini-monkey: Alleviating the semantic saw- tooth effect for lightweight mllms via complementary image pyramid, 2024

    Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alleviating the semantic saw- tooth effect for lightweight mllms via complementary image pyramid, 2024. 3, 4

    2024

  22. [30]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 2

    2019

  23. [31]

    Ku, Qian Liu, and Wenhu Chen

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi- image instruction tuning. Transactions on Machine Learning Research, 2024, 2024. 3, 4

    2024

  24. [32]

    Chatgpt for good? on opportuni- ties and challenges of large language models for education

    Enkelejda Kasneci, Kathrin Seßler, Stefan K ¨uchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, et al. Chatgpt for good? on opportuni- ties and challenges of large language models for education. Learning and individual differences, 103:102274, 2023. 1

    2023

  25. [33]

    Reflective decoding network for image captioning

    Lei Ke, Wenjie Pei, Ruiyu Li, Xiaoyong Shen, and Yu-Wing Tai. Reflective decoding network for image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8888–8897, 2019. 2

    2019

  26. [34]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 2, 3, 5

    2024

  27. [35]

    Building and better understanding vision- language models: insights and future directions., 2024

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions., 2024. 3, 4

    2024

  28. [36]

    What matters when building vision-language models?,

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?,

  29. [37]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 5, 6, 7, 8, 3, 4

    2024

  30. [38]

    Autobencher: Creating salient, novel, difficult datasets for language models

    Xiang Lisa Li, Evan Zheran Liu, Percy Liang, and Tat- sunori Hashimoto. Autobencher: Creating salient, novel, difficult datasets for language models. arXiv preprint arXiv:2407.08351, 2024. 1

    2024 arXiv

  31. [39]

    Llm-grounded video diffusion models, 2024

    Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Llm-grounded video diffusion models, 2024. 2

    2024

  32. [40]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 3, 4

    2023

  33. [41]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  34. [42]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1

    2024

  35. [43]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 3, 4

    2024

  36. [44]

    Tempcom- pass: Do video llms really understand videos?, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?, 2024. 2

    2024

  37. [45]

    Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024. 2

    2024

  38. [46]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 1

    2025

  39. [47]

    Mmdu: A multi-turn multi-image dia- log understanding benchmark and instruction-tuning dataset for lvlms, 2024

    Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, and Jiaqi Wang. Mmdu: A multi-turn multi-image dia- log understanding benchmark and instruction-tuning dataset for lvlms, 2024. 2

    2024

  40. [48]

    Mmalaya2

    DataCanvas Ltd. Mmalaya2. https://huggingface. co/DataCanvas/MMAlaya2, 2024. 3, 4

    2024

  41. [49]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. 2

    2024

  42. [50]

    Ovis: Structural em- bedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural em- bedding alignment for multimodal large language model. arXiv:2405.20797, 2024. 1, 5, 6, 7, 3, 4

    2024 arXiv

  43. [51]

    Mmlongbench-doc: Bench- marking long-context document understanding with visual- izations, 2024

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. Mmlongbench-doc: Bench- marking long-context document understanding with visual- izat...

    2024

  44. [52]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024. 1, 3, 4

    2024

  45. [53]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 1, 2, 5, 6, 7, 3, 4

    2024

  46. [54]

    Gpt-4 technical report

    R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023. 1

    2023 arXiv

  47. [55]

    Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annota- tions, 2024

    Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongy- ing Tu, and Conghui He. Omnidocbench: Benchmarking diverse pdf ...

    2024

  48. [56]

    Sowing information: Cultivating con- textual coherence with mllms in image generation, 2024

    Yuhan Pei, Ruoyu Wang, Yongqi Yang, Ye Zhu, Olga Rus- sakovsky, and Yu Wu. Sowing information: Cultivating con- textual coherence with mllms in image generation, 2024. 2

    2024

  49. [57]

    Rbdash-v1.2-72b

    RBDash-Team. Rbdash-v1.2-72b. https : / / huggingface . co / RBDash - Team / RBDash - v1.2-72b, 2024. 6, 7, 8, 3, 4

    2024

  50. [58]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2 10

    2022

  51. [59]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders, 2025

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catan- zaro, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders, 2025. 3, 4

    2025

  52. [60]

    Journeydb: A benchmark for generative im- age understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative im- age understanding. Advances in Neural Information Process- ing Systems, 36, 2024. 4

    2024

  53. [61]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  54. [62]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint,

  55. [63]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv...

    2024 arXiv

  56. [64]

    Llama-3-mixsensev1 1

    Zero Vision. Llama-3-mixsensev1 1. https : / / huggingface . co / Zero - Vision / Llama - 3-MixSenseV1_1, 2024. 3, 4

    2024

  57. [65]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  58. [66]

    Large-scale multi-modal pre-trained models: A comprehensive survey

    Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research , 20 (4):447–482, 2023. 1

    2023

  59. [67]

    WeMM. Wemm. https : / / github . com / scenarios/WeMM, 2024. 3, 4

    2024

  60. [68]

    Bloomberggpt: A large language model for finance

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, et al. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023. 1

    2023 arXiv

  61. [69]

    Unigen: A unified framework for textual dataset generation using large language models

    Siyuan Wu, Yue Huang, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xiangliang Zhang, Jianfeng Gao, Chaowei Xiao, et al. Unigen: A unified framework for textual dataset generation using large language models. arXiv preprint arXiv:2406.18966, 2024. 1

    2024

  62. [70]

    Self-correcting llm-controlled diffu- sion models

    Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6327– 6336, 2024. 2

    2024

  63. [71]

    XinYuan. Xinyuan. https://huggingface.co/ Cylingo/Xinyuan-VL-2B, 2024. 3, 4

    2024

  64. [72]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. 2, 1

    2023 arXiv

  65. [73]

    xgen-mm (blip-3): A family of open large multimodal models, 2024

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models, 2024. 6, 7, 8, 3, 4

    2024

  66. [74]

    Cc-ocr: A comprehensive and challenging ocr benchmark for evalu- ating large multimodal models in literacy, 2024

    Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Shuai Bai, LianWen Jin, and Junyang Lin. Cc-ocr: A comprehensive and challenging ocr benchmark for evalu- ating large multimodal models in literacy, 2024. 2

    2024

  67. [75]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6, 7, 8, 3, 4

    2024 arXiv

  68. [76]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingn- ing Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural In- formation Processing Systems, 36, 2024. 2, 1

    2024

  69. [77]

    Benchmarking chinese text recognition: Datasets, baselines, and an empirical study, 2022

    Haiyang Yu, Jingye Chen, Bin Li, Jianqi Ma, Mengnan Guan, Xixi Xu, Xiaocong Wang, Shaobo Qu, and Xiangyang Xue. Benchmarking chinese text recognition: Datasets, baselines, and an empirical study, 2022. 2

    2022

  70. [78]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 1

    2023 arXiv

  71. [79]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023. 2, 1

    2023

  72. [80]

    Task me anything

    Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. arXiv preprint arXiv:2406.11775, 2024. 2, 1

    2024 arXiv

  73. [81]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 1

    2023 arXiv

  74. [82]

    Omchat: A recipe to train multimodal language models with strong long context and video under- standing, 2024

    Tiancheng Zhao, Qianqian Zhang, Kyusong Lee, Peng Liu, Lu Zhang, Chunxin Fang, Jiajia Liao, Kelei Jiang, Yibo Ma, and Ruochen Xu. Omchat: A recipe to train multimodal language models with strong long context and video under- standing, 2024. 3, 4

    2024

  75. [83]

    Dyval: Dynamic evalua- tion of large language models for reasoning tasks

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evalua- tion of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representa- tions, 2023. 1

    2023

  76. [84]

    role”, “definition

    Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. In Forty-first International Conference on Machine Learning, 2024. 1 11 MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image G...

    2024

  77. [86]

    image pattern

    Annotate possible patterns in the image based on the image elements and the description of the image, and explain the reasoning for each identified pattern. # Key Points Carefully inspect all details within the image and annotate possible patterns for the image. Patterns shoul...

  78. [87]

    Summarize the optimal pattern list based on the input JSON data (including image patterns and frequencies)

  79. [88]

    Surreal”: 2262, “Lighting

    Describe the explanations for summarizing the pattern list. # Key Points Carefully examine the image patterns and their frequencies in the input data, summarize a new list of image patterns, and provide reasons. The new image pattern words should be as short as possible. Not o...

  80. [89]

    Generate a detailed description of the image based on its content, including all details observed in the image

  81. [90]

    # Key Points

    Annotate possible patterns in the image based on the image elements and the description of the image, and explain the reasoning for each identified pattern. # Key Points

  82. [91]

    Carefully inspect all details within the image and annotate possible patterns for the image

  83. [92]

    You may annotate multiple patterns as appropriate

    Patterns should be based on the contents and visual elements of the image. You may annotate multiple patterns as appropriate

  84. [93]

    Surreal”: “This pattern is characterized by its prevalence in depicting scenes that mix elements of fantasy with reality, often creating imaginative or dream-like visuals

    The labeling pattern can only come from the following patterns: { “Surreal”: “This pattern is characterized by its prevalence in depicting scenes that mix elements of fantasy with reality, often creating imaginative or dream-like visuals.”, “Technology”: “Highlights themes rel...

  85. [94]

    Accurately understand the input image and precisely generate an image caption-prompt

  86. [95]

    The generated image caption-prompt, when provided to the Text-to-Image model, requires the Text-to-Image model to generate a new image that is as consistent as possible with the input image

  87. [96]

    The generated image caption-prompt must conform to the preferences of the Text-to-Image model

  88. [97]

    BHNORAK TOP

    The generated image caption-prompt should describe the input image in as much detail as possible, and it should be between 20 to 60 words. # Output Format A string, that is the image caption-prompt. No extra output needed. Figure 16. Evaluation Pipeline Prompt. Text of Multi-m...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.