Pith. sign in

REVIEW 5 major objections 8 minor 1 cited by

FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation

T0 review · 5 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlagEvalMM is an open-source framework that evaluates multimodal models that both understand images and generate them, separating the model's inference step from the scoring step so new models and tasks can be plugged in without rewriting…

desk verdict A genuinely useful open-source evaluation harness for both understanding and generation, but the paper's empirical claims rest on unreleased datasets and an undisclosed ranking formula. read the letter →

arxiv 2506.09081 v3 pith:VDVWZJ5I submitted 2025-06-10 cs.CV cs.AIcs.CL

Zheqi He , Yesheng Liu , Jing-shu Zheng , Xuejing Li , Jin-Ge Yao , Bowen Qin , Richeng Xuan , Xi Yang This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords multimodalevaluationframeworkvision-languageunderstandingtext-to-imagegenerationtext-to-videomodeldecouplingleaderboardhumaninferenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multimodal evaluation can be unified: one open-source framework, FlagEvalMM, assesses vision-language understanding, text-to-image and text-to-video generation, and image-text retrieval on a single platform. Its central architectural proposal is to decouple model inference from scoring, letting each run in its own environment and communicate over a lightweight HTTP protocol, which the authors argue resolves the environment conflicts and rigidity of existing single-task toolkits. The demonstration ranks more than fifty understanding models and thirty generation models, reporting that open-source Qwen2.5-VL-72B outperforms several earlier commercial models, that some models show a sharp English-versus-Chinese performance gap, and that automated text-to-image metrics agree with human judges only moderately. A sympathetic reader would care because a working decoupled framework would make benchmark comparisons cheaper, more flexible, and unified across understanding and generation, with the reported metric-versus-human gaps pointing at where automated evaluation still needs work.

What carries the argument

The central object is the evaluation server/model runner split joined by a five-endpoint RESTful protocol: a Task on the server bundles a Processor (data normalization), a Config (prompt templates and metrics), and an Evaluator (scoring), while the runner's Model Adapter fetches data and submits results and its Backend executes inference under vLLM, SGLang, LMDeploy, Transformers, or Diffusers. The load-bearing simplification is that every evaluation lifecycle step maps to one API call, so the two sides never share a runtime. A SQLite cache keyed by a hash of input text, images, and parameters makes repeated or partially re-run evaluations avoid recomputation, and the plugin registry is what converts 'add a task' from code surgery into configuration.

What would settle it

Run a fixed panel of ten models through FlagEvalMM and through an independent framework (for example Lmms-Eval for understanding and VBench for generation) on the overlapping public datasets, and check that scores and rank orders coincide; then release the 414 T2I prompts and the self-constructed VQA items and have a second, independent annotation team rescore the same generated images — if rankings flip or agreement between frameworks drops well below the reported level, the framework's demonstrated accuracy claim would not be reproducible.

Watch

Extended reading notes

Core claim

FlagEvalMM is presented as a unified evaluation platform whose defining design choice is separating the model runner from the evaluation server: the runner only executes inference through pluggable backends and adapters, the server only distributes data, stores configs, and computes metrics, and the two sides communicate through a short RESTful protocol (get_tasks, task_info, get_meta, get_data, submit). The authors claim this decoupling makes environment conflicts disappear, lets inference and scoring scale independently, and lets new tasks and models register as plugins without touching core code, while a SQLite hash-cache and vLLM/SGLang backends keep evaluation fast. The empirical case rests on the framework's own leaderboard: in VLM ranking, Qwen2.5-VL-72B lands ahead of several earlier commercial models, while Mistral-Small-3.1 and Claude 3.7 show noticeably better English than Chinese scores; in text-to-image ranking, commercial models lead and the automatic metrics diverge from human judgment, with VQAScore correlating at 0.76 with human consistency scores and OneAlign-Aesthetic at 0.59 with human aesthetic scores.

Load-bearing premise

The 'accurate insights' claim rests on the unpublished self-constructed datasets: if the 414 text-to-image prompts and the subjective VQA and text-recognition items are unrepresentative or biased, the leaderboard rankings and capability profiles would not generalize, and the paper gives no external validation of them.

Editorial extensions

If this is right

  • A lab can run the same model through understanding benchmarks (MMMU, MathVerse, Blink) and generation benchmarks (COCO, GenAI-Bench, custom prompts) without switching frameworks or rewriting code.
  • Because inference and scoring run separately, heavy GPU serving (vLLM, SGLang) can be shared across experiments while evaluation happens on cheaper instances, cutting the cost of large leaderboard runs.
  • The hash-based cache makes re-evaluation nearly free: re-running a task after changing the prompt or metric reuses stored inference results instead of re-invoking the model.
  • The reported correlations imply that any single automated metric is a weak proxy for human judgment of generated images, so trustworthy generation leaderboards should combine several metrics with human scoring.
  • The observed English-Chinese gaps and the spatial-reasoning failures give model developers specific targets: cross-lingual generalization and perception under occlusion and perspective change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the decoupled protocol were adopted as a shared standard rather than a single project's interface, one evaluation server could feed many independent model runners, making leaderboard scores reproducible across labs — an outcome the paper does not claim but its design invites.
  • The 0.59 aesthetic correlation is the weakest link in the automated pipeline; a natural follow-up is to test whether newer learned metrics, including vision-language judges, close that gap, which this paper does not run.
  • The framework could be pointed inward: the same task and evaluator machinery could monitor a model during fine-tuning, not just rank finished models, since the decoupling makes mid-training checks cheap.
  • Because the self-constructed subsets are not released, an independent reproduction of the two leaderboard tables is currently impossible; releasing the 414 prompts and the subjective VQA items would settle whether the rankings generalize beyond the authors' own annotation team.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces FlagEvalMM, an open-source framework for evaluating multimodal models across understanding and generation tasks, including VQA, text-to-image/video generation, and image-text retrieval. The main architectural contribution is a decoupled design that separates model inference (Model Runner) from evaluation (Evaluation Server), communicating via a lightweight HTTP protocol. The framework supports acceleration backends such as vLLM and SGLang, provides plugin-based extensibility, and is integrated with the FlagEval platform and Hugging Face Spaces. The paper presents leaderboard results for selected VLM and T2I models, claiming that FlagEvalMM offers accurate and efficient insights into model strengths and limitations.

Significance. If the framework works as described, it is a useful practical contribution: the decoupled architecture, modular task registration, caching mechanism, and support for multiple inference backends are genuine conveniences for the evaluation community, and the open-source release is a concrete asset. However, the empirical evidence supporting the advertised 'accurate and efficient insights' is thin: there are no efficiency measurements, no uncertainty estimates on leaderboard scores, no inter-annotator agreement statistics for the human evaluation, and the self-constructed datasets that drive the leaderboard are not released or validated. These gaps currently limit the scientific claims that can be drawn from the paper, while the software contribution itself remains plausible and valuable.

major comments (5)
  1. [Section 4.2.2, Table 2] The T2I ranking is defined as a 'weighted average of human evaluation scores,' but the weights are never specified in the paper. Section 4.1.2 only states that the final score per aspect is the average of three human scores, and Appendix D provides the rubric but no aggregation formula. Because the ranking of close models (e.g., DALL-E 3, Kolors, and FLUX.1 schnell) could change under different plausible weightings, the headline leaderboard is not reproducible from the information given. This is a load-bearing omission for the claim that the framework provides accurate insights.
  2. [Sections 4.1.1 and 4.1.2] The leaderboard heavily relies on self-constructed evaluation datasets: a subjective image-text QA dataset, a text recognition dataset, 414 self-designed T2I prompts, and 100 self-designed T2V prompts. These datasets are not released, their construction or sampling procedure is not described, and no validation against external benchmarks is provided. The 'accurate' evaluation claim is therefore not independently verifiable, and the possibility of biased or unrepresentative prompts cannot be excluded. At minimum, the paper should release the prompts or provide a detailed description and a comparison with public benchmarks.
  3. [Abstract and Section 1] The paper claims that FlagEvalMM 'significantly enhance[s] evaluation efficiency' through vLLM, SGLang, and asynchronous data loading, but no efficiency measurements, throughput numbers, or comparisons against other frameworks are reported anywhere in the paper. This efficiency claim is load-bearing for the abstract and introduction, and it currently rests on architectural assertions rather than empirical evidence. A basic benchmark measuring end-to-end evaluation time on a fixed task would be necessary to support the claim.
  4. [Section 4.1.2 and Appendix D] Human evaluation is conducted with only three annotators, and although Appendix D mentions repeated rounds to ensure stability, no inter-annotator agreement statistics (e.g., Cohen's kappa or Krippendorff's alpha) are reported. Given that the T2I leaderboard is determined by a weighted average of human scores and that the score differences among adjacent models are small (e.g., DALL-E 3 at 70.12 vs. Kolors at 68.80), the absence of agreement or variance measures makes the precision of the ranking unjustified.
  5. [Section 4.2.1, Table 1] The VLM leaderboard's 'Average Rank' is not defined: it is unclear over which set of benchmarks and capabilities the ranks are averaged, how the language-specific ranks are combined, or what the 'Overall' column represents. The table shows 'EN' and 'ZH' rank columns but does not explain how they are aggregated into the overall rank. Without this definition, the headline ordering of VLMs is not interpretable or reproducible.
minor comments (8)
  1. [Section 1] There is a duplicated word in the sentence 'it can can also impede efficient resource usage'; this should be corrected.
  2. [Section 4.1.1] The text says 'dach dataset can be mapped to one or more capabilities'; this should read 'each dataset'.
  3. [Section 4.1.1] The phrase 'we have evaluate more than 50 multimodal understanding models' should be 'we have evaluated'.
  4. [Appendix A] The list of supported APIs mentions 'Hunyan; this is likely a typo for 'Hunyuan'.
  5. [Table 2] The automated evaluation columns are labeled 'VQAS', 'OA-Qua', and 'OA-Aes' but the text refers to 'VQAScore' and 'OneAlign-Aesthetic'; the abbreviations should be defined in the table caption or the accompanying text.
  6. [References] The reference to 'Huang et al., 5555' has an invalid year; it should be corrected to the actual publication year of T2I-CompBench++.
  7. [Section 3.2] The backend list mentions 'MLDeploy' in one place and 'LMDeploy' in another; the spelling should be consistent.
  8. [Section 4.1.2] The paper states the T2V dataset includes '148 prompts (100 self-designed, 48 public)' in Section 4.1.2 but the abstract and introduction only mention 100 self-designed prompts for the leaderboard; the total count and composition should be clarified consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework paper makes architectural and empirical claims, and the leaderboard results are not equivalent to their inputs by construction.

full rationale

This paper is a system and evaluation framework paper rather than a derivation. The central claims are architectural (decoupling inference from evaluation, modular task integration, inference acceleration) and are supported by deployment descriptions and experiments, not by a mathematical chain that reduces a predicted quantity to a fitted input. The VLM leaderboard in Table 1 relies mainly on external public benchmarks (MMMU, MathVision, MathVerse, Blink, Charxiv, CMMMU, MMVET-v2, CII-Bench), which provide independent grounding. The two self-constructed VLM datasets and the self-designed T2I prompts are evaluation inputs, not results derived from the framework; scoring them does not make the framework's conclusion equivalent to its inputs by construction. The only self-citation identified, CMMU (He et al., 2024b), appears as one benchmark among many and is not load-bearing for the framework's central claim. The T2I ranking uses an unspecified weighted average of human evaluation scores and unreleased prompt sets, which is a transparency and reproducibility limitation rather than a circularity: the ranking still depends on collected human judgments and public automated metrics, and there is no shown equation or fitted parameter that forces the reported conclusion independently of data. Because no specific reduction of a claimed result to its own inputs can be exhibited, the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 4 invented entities

The central claim rests on assumptions about benchmark validity, unstated ranking weights, and the reliability of a small human evaluation panel; these are not derived in the paper.

free parameters (1)
  • T2I ranking weighting scheme
    Section 4.2.2 ranks models by a weighted average of human evaluation scores without stating the weights or how they were chosen; the ranking order depends on these unstated values.
assumptions (3)
  • domain assumption Public benchmarks and their default prompts and accuracy calculations provide valid measures of multimodal capability.
    Invoked in Section 4.1.1 when scores are aggregated across Charxiv, CMMMU, MMMU, MathVision, and others; no psychometric validation is offered.
  • ad hoc to paper Self-constructed datasets and prompts are unbiased and representative enough to rank models.
    Sections 4.1.1 and 4.1.2 introduce self-constructed VQA, text, T2I, and T2V evaluation sets without release links, construction details, or validation evidence.
  • domain assumption Three annotators' averaged scores are a stable ground truth for ranking text-to-image models.
    Section 4.1.2 and Appendix D describe three evaluators scoring in repeated rounds, but no inter-annotator agreement statistic is reported.
invented entities (4)
  • Self-constructed VQA dataset
    purpose: Measure general knowledge and visual perception in English and Chinese.
    No release location, sample items, or validation against existing benchmarks are given in the paper.
  • Self-constructed text recognition dataset
    purpose: Measure text recognition and understanding.
    Described only as self-constructed in Section 4.1.1; no data or annotation details beyond the OCR-Bench-like string matching.
  • Self-constructed T2I prompt set (414 prompts)
    purpose: Rank text-to-image models on consistency, realism, aesthetics, and safety.
    Mentioned in Section 4.1.2 with no release and no demonstration that the prompts are a representative sample.
  • Self-constructed T2V prompt set (100 prompts)
    purpose: Evaluate text-to-video generation.
    Section 4.1.2 states 100 self-designed and 48 public prompts but provides no access or examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation." pith.science (2026). https://pith.science/paper/VDVWZJ5I

@misc{pith2026250609081,
  author       = {Pith},
  title        = {Pith review of: FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDVWZJ5I}},
  note         = {Machine review of arXiv:2506.09081}
}
read the original abstract

We present FlagEvalMM, an open-source evaluation framework designed to comprehensively assess multimodal models across a diverse range of vision-language understanding and generation tasks, such as visual question answering, text-to-image/video generation, and image-text retrieval. We decouple model inference from evaluation through an independent evaluation service, thus enabling flexible resource allocation and seamless integration of new tasks and models. Moreover, FlagEvalMM utilizes advanced inference acceleration tools (e.g., vLLM, SGLang) and asynchronous data loading to significantly enhance evaluation efficiency. Extensive experiments show that FlagEvalMM offers accurate and efficient insights into model strengths and limitations, making it a valuable tool for advancing multimodal research. The framework is publicly accessible at https://github.com/flageval-baai/FlagEvalMM.

Figures

Figures reproduced from arXiv: 2506.09081 by the authors.

Figure 1
Figure 1. Framework of FlagEvalMM multimodal models capable of integrating both un￾derstanding and generation functionalities (Chen et al., 2025; Wang et al., 2024b). These developments underscore the need for ef￾ficient and comprehensive evaluation frameworks assess multimodal models’ diverse capabilities. An ideal evaluation framework should accurately, effi￾ciently, and conveniently assess various capabilities across diver… view at source ↗
Figure 2
Figure 2. Components and workflow of the evaluation server [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Communication protocol between evaluation server and model runner [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoboBrain 2.0 Technical Report

    cs.RO 2025-07 conditional novelty 6.0 of 10

    RoboBrain 2.0, a 7B/32B embodied vision-language model built on Qwen2.5-VL, reports state-of-the-art or near-top scores on several spatial and temporal reasoning benchmarks for robotics.

Reference graph

Works this paper leans on

49 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mistral AI. 2025. https://mistral.ai/news/mistral-small-3-1 Mistral small 3.1

    work page 2025

  4. [4]

    Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-01-18

    work page 2024

  5. [5]

    Anthropic. 2025. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3-7-sonnet. Accessed: 2025-03-08

    work page 2025

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  7. [7]

    ByteDance. 2024. Doubao image. https://www.volcengine.com/docs/6791/1366783

    work page 2024

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024. Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

Show all 49 references
  1. [9]

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811

    2025 arXiv

  2. [10]

    Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing, pages 1--4

    2009

  3. [11]

    LMDeploy Contributors. 2023. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy

    2023

  4. [12]

    SGLang Contributors. 2024. https://github.com/sgl-project/sglang Sglang: A fast serving framework for large language models and vision language models . Accessed: 2025-03-23

    2024

  5. [13]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, and 1 others. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference ...

    2024

  6. [14]

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, and 1 others. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international confe...

    2024

  7. [15]

    Roy Thomas Fielding. 2000. Architectural styles and the design of network-based software architectures. University of California, Irvine

    2000

  8. [16]

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148--166. Springer

    2024

  9. [17]

    Kevin P Gaffney, Martin Prammer, Larry Brasfield, D Richard Hipp, Dan Kennedy, and Jignesh M Patel. 2022. Sqlite: past, present, and future. Proceedings of the VLDB Endowment, 15(12)

    2022

  10. [18]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  11. [19]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

    2024 arXiv

  12. [20]

    Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, and 1 others. 2024 a . Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. In Proceedings of the 2024 ...

    2024

  13. [21]

    Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. 2024 b . Cmmu: a benchmark for chinese multi-modal multi-type question understanding and reasoning. In Proceedings of the Thirty-Third International Joint Conference on Artificial I...

    2024

  14. [22]

    Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 5555. https://doi.ieeecomputersociety.org/10.1109/TPAMI.2025.3531907 T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation . IEEE Transactions on Pattern A...

    2025

  15. [23]

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench : Comprehensive benchmark suite for video generative mo...

    2024

  16. [24]

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, and 1 others. 2024. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603

    2024 arXiv

  17. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  18. [26]

    Black Forest Labs. 2024. Flux. https://github.com/black-forest-labs/flux

    2024

  19. [27]

    Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, and 1 others. 2024. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems, 37:140632--140666

    2024

  20. [28]

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, and 1 others. 2023. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36:69981--70011

    2023

  21. [29]

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and 1 others. 2024. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743

    2024 arXiv

  22. [30]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  23. [31]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, ...

    2014

  24. [32]

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366--384. Springer

    2024

  25. [33]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102

    2024

  26. [34]

    OpenAI. 2023. https://cdn.openai.com/papers/GPTV_System_Card.pdf Gpt-4v(ision) system card . OpenAI Research

    2023

  27. [35]

    OpenAI. 2024. https://openai.com/index/sora-system-card/ Sora

    2024

  28. [36]

    Qwen Team. 2025. https://arxiv.org/abs/2502.13923 Qwen2.5-vl technical report . Preprint, arXiv:2502.13923

    2025 arXiv

  29. [37]

    Tencent. 2024. Hunyuan image. https://cloud.tencent.com/document/product/1729/105969

    2024

  30. [38]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024 a . https://openreview.net/forum?id=QWTCcxMpPA Measuring multimodal mathematical reasoning with math-vision dataset . In The Thirty-eight Conference on Neural Information ...

    2024

  31. [39]

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, and 1 others. 2024 b . Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869

    2024 arXiv

  32. [40]

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, and 1 others. 2024 c . Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37...

    2024

  33. [41]

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, and 1 others. 2024. Q-align: teaching lmms for visual scoring via discrete text-defined levels. In Proceedings of the 41st International Conference on...

    2024

  34. [42]

    Weihao Yu, Zhengyuan Yang, Linfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. 2024. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.00765

    2024 arXiv

  35. [43]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024 a . Mmmu: A massive mul...

    2024

  36. [44]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2024 b . Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813

    2024 arXiv

  37. [45]

    Chenhao Zhang, Xi Feng, Yuelin Bai, Xinrun Du, Jinchang Hou, Kaixin Deng, Guangzeng Han, Qinrui Li, Bingli Wang, Jiaheng Liu, Xingwei Qu, Yifei Zhang, Qixuan Zhao, Yiming Liang, Ziqiang Liu, Feiteng Fang, Min Yang, Wenhao Huang, Chenghua Lin, and 2 others. 2024 a . https://arx...

    2024 arXiv

  38. [46]

    Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, and 1 others. 2024 b . Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944

    2024 arXiv

  39. [47]

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and 1 others. 2024 c . Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772

    2024 arXiv

  40. [48]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024 d . Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, page...

    2024

  41. [49]

    Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. Cogview3: Finer and faster text-to-image generation via relay diffusion. In European Conference on Computer Vision, pages 1--22. Springer

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.