Pith. sign in

REVIEW 4 major objections 5 minor 31 cited by

MiMo-VL Technical Report

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MiMo-VL-7B-RL, a 7-billion-parameter open vision-language model, beats Qwen2.5-VL-7B on 35 of 40 tasks, scores 59.4 on OlympiadBench, and sets a GUI-grounding high of 56.1 on OSWorld-G.

desk verdict MiMo-VL is a substantial model release with real open artifacts, but the self-reported benchmark sweep and image-only contamination control mean the headline numbers should be read as plausible claims, not settled facts. read the letter →

arxiv 2506.03569 v1 pith:GWIW4SQL submitted 2025-06-04 cs.CL

Xiaomi LLM-Core Team: Zihao Yue , Zhenru Lin , Yifan Song , Weikun Wang , Shuhuai Ren , Shuhao Gu , Shicheng Li , Peidian Li
show 65 more authors
Liang Zhao Lei Li Kainan Bao Hao Tian Hailin Zhang Gang Wang Dawei Zhu Cici Chenhong He Bowen Ye Bowen Shen Zihan Zhang Zihan Jiang Zhixian Zheng Zhichao Song Zhenbo Luo Yue Yu Yudong Wang Yuanyuan Tian Yu Tu Yihan Yan Yi Huang Xu Wang Xinzhe Xu Xingchen Song Xing Zhang Xing Yong Xin Zhang Xiangwei Deng Wenyu Yang Wenhan Ma Weiwei Lv Weiji Zhuang Wei Liu Sirui Deng Shuo Liu Shimao Chen Shihua Yu Shaohui Liu Shande Wang Rui Ma Qiantong Wang Peng Wang Nuo Chen Menghang Zhu Kangyang Zhou Kang Zhou Kai Fang Jun Shi Jinhao Dong Jiebao Xiao Jiaming Xu Huaqiu Liu Hongshen Xu Heng Qu Haochen Zhao Hanglong Lv Guoan Wang Duo Zhang Dong Zhang Di Zhang Chong Ma Chang Liu Can Cai Bingquan Xia
This is my paper
classification cs.CL
keywords vision-languagemodelmultimodalreasoningmixedon-policyreinforcementlearningchain-of-thoughtdataGUIgroundingpre-trainingrecipeopen-source7BGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a recipe-and-results report for MiMo-VL-7B, a 7-billion-parameter open-source vision-language model, and its central claim is that a model this size can beat much larger competitors on hard multimodal reasoning, perception, and GUI-grounding benchmarks if the training recipe is right. The recipe is: four pre-training stages over 2.4 trillion tokens, with large volumes of synthetic step-by-step reasoning data injected into the final stage, followed by a mixed on-policy reinforcement learning phase that optimizes rule-verifiable rewards and human-preference rewards at once. The reported results include a 35-of-40 win over Qwen2.5-VL-7B on the evaluated tasks, a 59.4 score on OlympiadBench that beats models with up to 78B parameters, and a 56.1 on OSWorld-G GUI grounding that beats the specialized UI-TARS model. The reason to care is that these are concrete, reproducible design choices rather than a bigger model: where to place reasoning data, why on-policy RL keeps improving where an older algorithm plateaus, and which frictions appear when reasoning, perception, and preference objectives are optimized together.

What carries the argument

The load-bearing machinery is the training recipe, built from two named components: a four-stage pre-training pipeline and MORL (Mixed On-policy Reinforcement Learning). The pipeline warms up a randomly initialized MLP projector on image-caption pairs (300B tokens), unfreezes the vision encoder for vision-language alignment (167B tokens), trains all parameters on general multimodal data covering OCR, grounding, video, and GUI traces (1.4T tokens), and finishes with a long-context stage at 32K sequence length in which long chain-of-thought reasoning data is injected at high proportion (550B tokens). MORL adapts GRPO into a fully on-policy rule that performs a single policy update per rollout with no clipped surrogate objective and no KL penalty, while a reward router dispatches rule-based rewards (exact-match for reasoning, GIoU or point-in-box for grounding, accuracy for counting, IoU for temporal video localization) and two Bradley-Terry reward models for human preference, all normalized to [0,1]. This machinery is doing the work of showing that long-CoT data can be absorbed during pre-training and that reasoning, perception, grounding, and preference objectives can be optimized in one pass, an integration the paper acknowledges is imperfect because reasoning tasks lengthen responses while grounding and counting tasks shorten them.

What would settle it

Search the released training corpora and the synthetic-reasoning datasets for near-duplicates of sampled questions from OlympiadBench, AIME 2024/2025, and MMMU-Pro using text-embedding or n-gram matching: if evaluation questions or close paraphrases appear in the training text, the 59.4 OlympiadBench and 52.5 AIME 2025 scores would reflect memorization rather than reasoning. A cleaner test would be to rerun the same four-stage recipe after text-level decontamination and check whether the reported 35-of-40 margin over Qwen2.5-VL-7B persists.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that MiMo-VL-7B-RL delivers state-of-the-art open-source results across the full spectrum of multimodal ability at 7B scale: it outperforms Qwen2.5-VL-7B on 35 of 40 evaluated tasks; it scores 59.4 on OlympiadBench, surpassing models with up to 78B parameters; and it scores 56.1 on OSWorld-G, setting the reported standard for GUI grounding above specialized interface agents. The paper traces these results to two findings. First, high-quality synthetic reasoning data with long chain-of-thought belongs in the later pre-training stages rather than in fine-tuning alone, and the paper reports that the final pre-training stage alone brought roughly +16 on OlympiadBench, +9 on MMMU, and +14 on OSWorld-G with no sign of saturation. Second, a fully on-policy variant of GRPO that mixes rule-verifiable rewards for reasoning, perception, grounding, counting, and temporal localization with two human-preference reward models improves nearly every benchmark, whereas the paper reports that vanilla GRPO plateaus around 20,000 samples.

Load-bearing premise

The headline scores assume the benchmarks measure true generalization rather than memorization: the paper's own deduplication compares images only, while several leading results come from text-heavy reasoning tests and the training data includes vast web, synthetic, and proprietary text that could overlap with evaluation questions.

Editorial extensions

If this is right

  • Moving long-CoT synthetic reasoning data into the final pre-training stages should transfer to other open models, since the paper reports sustained, non-saturating gains there: +16 on OlympiadBench, +9 on MMMU, and +14 on OSWorld-G in Stage 4 alone.
  • The paper's comparison of on-policy RL with vanilla GRPO implies that RL data budgets are better spent on fresh on-policy rollouts, because vanilla GRPO's early gains flatten near 20,000 samples while on-policy training keeps climbing.
  • The 56.1 OSWorld-G result implies a general-purpose 7B VLM can beat GUI-specialized models like UI-TARS on grounding tasks, suggesting dedicated agent training is not required for strong interface-grounded perception.
  • Because MORL improved performance on nearly every evaluated task despite reported inter-task interference, mixed-RL recipes with explicit length and difficulty controls should be able to push further than the current recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A text-level contamination audit is the natural next check: the paper's only overlap guard is image perceptual hashing, but several headline benchmarks are text-heavy reasoning sets and the corpus contains large web, synthetic, and proprietary text; near-duplicate matching could confirm or refute the 59.4 and 35-of-40 claims.
  • The OSWorld-G score is a no-refusal grounding accuracy on static screenshots rather than an end-to-end task-completion rate, so the GUI 'standard' may not carry over to full multi-step computer-use agents.
  • An ablation replacing the MiMo-7B backbone with a weaker base would isolate whether Stage 4's long-CoT gains come from the data or from a backbone already optimized for reasoning.
  • The Elo advantage rests on a style-controlled GPT-4o judge; if that judge favors longer reasoning traces, part of the RL model's user-preference win could be a response-length artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents MiMo-VL-7B, a vision-language model built from the MiMo-7B language model and a Qwen2.5-ViT encoder, trained in four pre-training stages (projector warmup, vision-language alignment, general multimodal pre-training, long-context SFT) consuming 2.4 trillion tokens, followed by a Mixed On-policy Reinforcement Learning (MORL) stage combining verifiable rewards, grounding, counting, temporal grounding, and RLHF. The central claims are that MiMo-VL-7B-RL outperforms Qwen2.5-VL-7B on 35 of 40 evaluated tasks, achieves 59.4 on OlympiadBench, and sets a new state of the art on OSWorld-G (56.1 in Table 2 and Figure 4, but 54.7 in Section 1). The authors open-source the model checkpoints and an evaluation suite covering 50+ tasks.

Significance. If the reported benchmark scores are accurate, the work is significant: it demonstrates that a 7B open-source VLM can match or exceed much larger models on several reasoning, perception, and GUI benchmarks, and it provides a concrete recipe (long-CoT reasoning data in pre-training and mixed on-policy RL) that the community can reproduce. The release of checkpoints and a full evaluation suite is a concrete contribution that will enable independent verification. However, the significance is conditional on resolving two load-bearing issues: the internal inconsistency in the OSWorld-G headline number, and the fact that the only stated decontamination is image-based phash deduplication, which does not address text leakage into text-heavy benchmarks such as OlympiadBench, AIME, and MMMU-Pro. The vague description of the evaluation harness also makes the exact '35/40' comparison hard to assess.

major comments (4)
  1. [Section 1 vs Table 2 / Figure 4] The OSWorld-G score is reported as 54.7 in Section 1 ('sets a new standard by achieving a score of 54.7 on OSWorld-G') but as 56.1 in Table 2 and Figure 4 for MiMo-VL-7B-RL. Since the abstract and introduction use this number as a headline claim, the discrepancy must be resolved and reconciled in all places, with a clear statement of which number is the final model's score and under which evaluation condition it was obtained.
  2. [Section 2.2 and Section 4.1] The only contamination-control measure described in Section 2.2 is 'phash-based image deduplication to eliminate potential overlaps between our training datasets and evaluation benchmarks.' This is an image-only check. Many of the benchmarks supporting the central claims are text-heavy (OlympiadBench, AIME24/25, MMMU-Pro, SuperGPQA), and the training mixture includes large amounts of web, interleaved, synthetic reasoning, and proprietary text data (Section 2.2.2, 2.2.6, 3.1). If evaluation text or near-duplicate text appears in those corpora, the reported scores would be inflated. The authors should provide text-level overlap statistics (e.g., n-gram or embedding-based contamination analysis) for the evaluation benchmarks against the training mixture, or a clear explanation of why text leakage is not a concern. This is load-bearing because the headline '35/40' and OlympiadBench claims depend on the scores measuring true generalization.
  3. [Section 4.1] The evaluation section states 'We further optimize the evaluation logic for specific tasks to ensure better evaluation consistency' without specifying what those optimizations are. Table 2 and Table 3 mark many baselines with an asterisk indicating they were run in the authors' framework. If the framework modifies prompts, decoding, answer extraction, or scoring relative to the official benchmark suites, small score differences could change the '35/40' count. The report should list, per task, the exact prompt template, decoding parameters (the text benchmarks use temperature 0.6 and top-p 0.95 while image benchmarks use greedy decoding, which is itself a relevant difference for reasoning tasks), answer extraction rules, and any task-specific 'optimizations', so that the comparisons are auditable.
  4. [Tables 2 and 3] Many of the score differences supporting the '35/40' claim are very small (e.g., MMBench-en 84.5 vs 84.4; AI2D 83.2 vs 83.5; DocVQA 95.2 vs 95.5 for a different comparison; WeMath 66.3 vs 65.1). No standard errors, confidence intervals, or multiple-run variance are reported for any benchmark. Given that the headline claim is a count of wins, the authors should provide a sensitivity analysis or error estimates (e.g., bootstrap or multiple evaluation seeds) to show that the 35/40 count is not an artifact of near-tie outcomes.
minor comments (5)
  1. [References] Bai et al. 2025a and Bai et al. 2025b refer to the same Qwen2.5-VL technical report; please cite it once and update all in-text references accordingly.
  2. [Section 4.2] The sentence beginning 'MiMo-VL-7B-SFT and(i)On general vision-language tasks...' is malformed and should be rewritten into clean prose.
  3. [Figure 1] The caption reads 'Figure 1Benchmark performance of MiMo-VL-7B.'; there should be a space after the figure number.
  4. [Section 2.2] The abbreviation 'phash' is used without expansion; please define it as 'perceptual hash' at first use.
  5. [Section 5.3] The claim that reasoning tasks encourage longer response lengths while grounding and counting tasks lead to shrinking lengths would be more convincing with a quantitative plot of average response length over the course of MORL training; consider adding it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central performance claims are measured against external third-party benchmarks, and MiMo-7B self-citations are predecessor inputs rather than load-bearing reductions.

full rationale

The paper's load-bearing claims are benchmark scores (e.g., 35/40 against Qwen2.5-VL-7B, 59.4 on OlympiadBench, 56.1 on OSWorld-G) obtained by evaluating trained models on external, third-party evaluation suites, not by fitting constants and then renaming them as predictions. The training recipe is a constructive pipeline: a base LLM (MiMo-7B) is initialized, multimodal pre-training data are curated, and post-training RL is applied; performance is then measured. The self-citations to Xiaomi (2025) supply the base model, the text-reasoning RL data, and the Seamless Rollout Engine, but these are inputs into the training process rather than conclusions that the evaluation is supposed to prove. The in-house GPT-4o-judged Elo rating is a preference-rank evaluation, not a derivation of the headline benchmark numbers. The image-only phash deduplication and the customized evaluation framework raise contamination and harness-consistency risks, but those are evidence-quality concerns, not circular reasoning: no equation, fitted parameter, or self-citation is shown to be equivalent by construction to the reported result. The evaluation scores are self-contained against external benchmarks, so no circular step meets the evidentiary bar.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. MORL and Reward-as-a-Service are names for engineering integrations of existing rewards, not new entities. The central claim rests on domain assumptions about benchmark validity, judge reliability, contamination control, and the correctness of the reward verifier.

free parameters (6)
  • Evaluation max image pixels = 4096 x 28 x 28
    Chosen by authors for image benchmarks; scores are conditional on this resolution cap. A different cap can change performance, and the choice is not derived from theory.
  • Maximum generation tokens = 32768
    Used for all long-CoT evaluations; could affect pass rates on reasoning benchmarks, especially with long reasoning traces.
  • Text evaluation sampling = temperature 0.6, top-p 0.95
    Used for text benchmarks with greedy search for image benchmarks; these choices affect reported scores.
  • Video sampling = 2 FPS, max 256 frames, 16384 tokens
    Video benchmark protocol chosen by the authors; different sampling rates change video understanding scores.
  • Training learning rates = 1e-3, 1e-4, 1e-5, 2.5e-5
    Stage-wise learning rates are hand-chosen and central to the training recipe's success claims.
  • Training sequence lengths = 8K, 8K, 8K, 32K
    Sequence length schedule is a design choice; Stage 4 extends to 32K for long context.
assumptions (6)
  • domain assumption The benchmarks used are valid proxies for general visual understanding, multimodal reasoning, GUI grounding, and user preference.
    The paper equates high scores on MMMU, OlympiadBench, OSWorld-G, etc., with the claimed capabilities; if these benchmarks are saturated or misaligned, the 'state-of-the-art' claims weaken.
  • domain assumption GPT-4o-based evaluation for dagger benchmarks and Elo is reliable and unbiased for the compared models.
    Many scores in Tables 2 and 3 and the Elo ratings rely on GPT-4o as judge; no validation of judge agreement is reported.
  • domain assumption phash-based image deduplication and text filtering are sufficient to prevent evaluation contamination.
    Section 2.2 states this is used to 'minimize contamination,' but the mechanism is described only for images, and no evidence is given that text overlap is controlled.
  • domain assumption The Math-Verify library correctly determines correctness for all rule-based rewarded tasks.
    RLVR rewards and several evaluations rely on this library; an incorrect verifier would corrupt both training signal and reported reasoning scores.
  • domain assumption MiMo-7B-Base, the authors' prior model, is a strong reasoning backbone and its properties transfer to multimodal training.
    The paper cites Xiaomi (2025) for the base model and the on-policy RL recipe; this is self-reliance on a predecessor rather than an independently established theorem.
  • domain assumption The Seamless Rollout Engine and Reward-as-a-Service infrastructure perform as described without hidden reward shaping.
    Training outcome claims depend on the RL infrastructure, but no implementation details or code are provided to verify the absence of additional reward signals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiMo-VL Technical Report." pith.science (2026). https://pith.science/paper/GWIW4SQL

@misc{pith2026250603569,
  author       = {Pith},
  title        = {Pith review of: MiMo-VL Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWIW4SQL}},
  note         = {Machine review of arXiv:2506.03569}
}
read the original abstract

We open-source MiMo-VL-7B-SFT and MiMo-VL-7B-RL, two powerful vision-language models delivering state-of-the-art performance in both general visual understanding and multimodal reasoning. MiMo-VL-7B-RL outperforms Qwen2.5-VL-7B on 35 out of 40 evaluated tasks, and scores 59.4 on OlympiadBench, surpassing models with up to 78B parameters. For GUI grounding applications, it sets a new standard with 56.1 on OSWorld-G, even outperforming specialized models such as UI-TARS. Our training combines four-stage pre-training (2.4 trillion tokens) with Mixed On-policy Reinforcement Learning (MORL) integrating diverse reward signals. We identify the importance of incorporating high-quality reasoning data with long Chain-of-Thought into pre-training stages, and the benefits of mixed RL despite challenges in simultaneous multi-domain optimization. We also contribute a comprehensive evaluation suite covering 50+ tasks to promote reproducibility and advance the field. The model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-VL.

Discussion (0). Sign in to comment.

Forward citations

Cited by 31 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Across 26 LVLMs on MemeBench, all models show a 14.6-29.0% gap between visual coverage and cultural-knowledge coverage, and retrieval raises knowledge while lowering visual coverage.

  2. TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    TimeLens2 shows that a compact video MLLM can localize multiple evidence intervals in long videos by training on verified interval labels and a Wasserstein-based time-distance reward.

  3. Learning Self-Correction in Vision-Language Models via Rollout Augmentation

    cs.CV 2026-02 conditional novelty 7.0 of 10

    Recombining pre/post-correction segments of existing rollouts yields dense self-correction training signals, giving an 8B VLM a 1.0-point average accuracy gain over GSPO on 7 benchmarks at 0.72x rollout time.

  4. VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?

    cs.CV 2026-02 conditional novelty 7.0 of 10

    VISTA-Bench shows vision-language models degrade on visualized text in images compared to equivalent pure text, with larger gaps under increased perceptual difficulty.

  5. Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    Video-OPD uses on-policy distillation from a frontier teacher to turn sparse episode rewards into dense step-wise signals for more efficient post-training of MLLMs on temporal video grounding.

  6. Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Models

    cs.CR 2025-08 conditional novelty 7.0 of 10

    Hidden Tail crafts adversarial images that force VLMs to emit long invisible runs of special tokens, inflating output length up to 19.2x while keeping the visible answer normal.

  7. HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

    cs.IR 2026-07 conditional novelty 6.5 of 10

    A hierarchical evidence-graph benchmark reveals that multimodal deep-research models write fluent reports while failing citation, claim, and answer grounding.

  8. Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools

    cs.CV 2026-07 accept novelty 6.5 of 10

    Self-regulated implicit visual tools let MLLMs match or beat external-tool and latent-reasoning methods on high-resolution perception while cutting inference latency by up to 86%.

  9. CAVE: Competence-Aware Visual Boundary Evidence Alignment for Video Temporal Grounding

    cs.CL 2026-08 conditional novelty 6.0 of 10

    CAVE adds boundary evidence tokens and an attention alignment reward to RL-based video temporal grounding, improving average timestamp accuracy by about 1.9 points over TimeLens-7B.

  10. RefCaptioner: Multi-Reference Image-Grounded Video Captioning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Mixed-data SFT plus Hierarchical Coverage-Discounted GRPO yields open-source SOTA multi-reference image-grounded video captions on MRVBench without hurting general captioning.

  11. PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis

    cs.CV 2026-07 conditional novelty 6.0 of 10

    PathScale-R1 and PathScale-VQA raise cross-scale pathology VQA accuracy to 83% by training on expert multi-magnification paths with anti-shortcut curation and scale-aware RL.

  12. MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ESJ statement verification, the INSETS-462k corpus, the MVEI benchmark, and the EmObserver model together provide a scalable way to evaluate and improve MLLM visual emotional intelligence.

  13. Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    SIS-Bench shows MLLMs lag on UAV self-awareness versus spatial tasks, and motion-aware optical-flow fusion improves perception and memory.

  14. NEXT: Reasoning-Driven Video Recommendation via a Vision-Language Model

    cs.IR 2026-06 conditional novelty 6.0 of 10

    NEXT, a reasoning-driven video recommendation framework with a custom-trained 8B VLM, yields +0.53% watch time and +0.51% distinct exposure in production A/B tests.

  15. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.

  16. VLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality

    cs.CV 2025-09 conditional novelty 6.0 of 10

    VLM-in-the-Wild provides an enterprise-focused benchmark and the BlockWeaver OCR matching algorithm, reporting that a small fine-tuned model can rival a 32B model on some tasks.

  17. Kwai Keye-VL 1.5 Technical Report

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Keye-VL-1.5 combines similarity-based Slow-Fast video token allocation with progressive context extension and iterative RL, reporting leading video-understanding results among 8B-scale multimodal models.

  18. LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

    cs.CV 2025-08 conditional novelty 6.0 of 10

    RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...

  19. R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    R-4B uses bi-mode annealing plus a GRPO variant with forced thinking and non-thinking rollouts so a 4B multimodal model adaptively picks when to reason, reportedly matching 16B thinking models on several reasoning benchmarks.

  20. EMIT: Enhancing MLLMs for Industrial Anomaly Detection via Difficulty-Aware GRPO

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A difficulty-aware GRPO training scheme with response resampling, advantage reweighting, GPT-generated text samples, and heatmap-guided contrastive embeddings improves InternVL3-8B by 7.77 percentage points on the MMA...

  21. Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

    cs.CV 2025-07 conditional novelty 6.0 of 10

    RRVF trains an image-to-code MLLM using reinforcement learning with a render-and-compare visual feedback loop, and it outperforms supervised fine-tuning on chart and web benchmarks.

  22. MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.

  23. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  24. VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.

  25. OPD-V: Visual On-Policy Self-Distillation with Modality Balance

    cs.CV 2026-08 conditional novelty 5.0 of 10

    OPD-V selects self-distillation tokens by comparing a zoomed-in teacher against a masked-image teacher, improving MLLM visual reasoning while reducing training cost.

  26. Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation

    cs.AI 2026-02 reject novelty 5.0 of 10

    A capability-oriented multimodal judge benchmark and MCTS-based preference-data generation improve judge models on some benchmarks, but the paper's SOTA claim is not supported by its own numbers.

  27. Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning

    cs.CV 2025-10 conditional novelty 5.0 of 10

    HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...

  28. VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

    cs.CL 2025-10 conditional novelty 5.0 of 10

    EVisRAG, an evidence-guided observe-record-reason-answer pipeline trained with reward-scoped GRPO, improves multi-image visual QA accuracy by about 19% over its backbone VLM across five benchmarks.

  29. MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A new benchmark of 2,000 K-12 math questions photographed in real scenes finds that multimodal LLMs perform significantly worse on authentic handheld images than on clean benchmarks.

  30. Skywork-R1V3 Technical Report

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 38B open-source VLM reaches 76.0% on MMMU using RL post-training and connector-only tuning, with a critical-token entropy metric for checkpoint selection.

  31. How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study

    cs.CV 2025-07 reject novelty 4.0 of 10

    A ten-model, seven-benchmark medical VLM evaluation whose headline reasoning-vs-understanding finding is contradicted by its own tables.

Reference graph

Works this paper leans on

75 extracted references · 29 canonical work pages · cited by 31 Pith papers

  1. [1]

    Alayrac, J

    J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a visual language model...

    work page 2022

  2. [3]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025 b

    arXiv 2025

  3. [4]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. 0: A vision-language-action flow model for general robot control. ArXiv, abs/2410.24164, 2024....

    arXiv

  4. [5]

    L. Chen, L. Li, H. Zhao, Y. Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than \ 3. https://github.com/Deep-Agent/R1-V, 2025 a . Accessed: 2025-02-02

    work page 2025

  5. [6]

    X. Chen, Z. Zhao, L. Chen, D. Zhang, J. Ji, A. Luo, Y. Xiong, and K. Yu. Websrc: a dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465, 2021

    arXiv 2021

  6. [7]

    Y. Chen, Z. Yang, Z. Liu, C. Lee, P. Xu, M. Shoeybi, B. Catanzaro, and W. Ping. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400, 2025 b

    arXiv 2025

  7. [8]

    Cheng, Q

    K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li, J. Zhang, and Z. Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    arXiv 2024

  8. [9]

    C. Chou, L. Dunlap, K. Mashita, K. Mandal, T. Darrell, I. Stoica, J. Gonzalez, and W.-L. Chiang. Visionarena: 230k real world user-vlm conversations with preference labels. ArXiv, abs/2412.08687, 2024. URL https://api.semanticscholar.org/CorpusID:274655992

    work page Pith review arXiv 2024

Show all 75 references
  1. [10]

    W. Dai, N. Lee, B. Wang, Z. Yang, Z. Liu, J. Barker, T. Rintamaki, M. Shoeybi, B. Catanzaro, and W. Ping. Nvlm: Open frontier-class multimodal llms. arXiv preprint, 2024

    2024

  2. [11]

    Deitke, C

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y. Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. ArXiv preprint, abs/2409.17146, 2024

    2024 arXiv

  3. [12]

    X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. ArXiv preprint, abs/2502.14739, 2025. URL https://arxiv.org/abs/2502.14739

    2025 arXiv

  4. [13]

    D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North A merican Chapter of th...

    2019 doi

  5. [14]

    C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024 a

    2024 arXiv

  6. [15]

    X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148--166. Springer, 2024 b

    2024

  7. [16]

    J. Gao, C. Sun, Z. Yang, and R. Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267--5275, 2017

    2017

  8. [17]

    C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024

    2024 arXiv

  9. [18]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv preprint, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  10. [19]

    K. Hu, P. Wu, F. Pu, W. Xiao, Y. Zhang, X. Yue, B. Li, and Z. Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  11. [20]

    Jiang, X

    D. Jiang, X. He, H. Zeng, C. Wei, M. Ku, Q. Liu, and W. Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  12. [21]

    Karamcheti, S

    S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In ICML, 2024

    2024

  13. [22]

    Kazemzadeh, V

    S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787--798, 2014

    2014

  14. [23]

    Kembhavi, M

    A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pages 235--251. Springer, 2016

    2016

  15. [24]

    Lambert, J

    N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi. Tulu 3: Pushin...

    2025 arXiv

  16. [25]

    B. Li, Y. Ge, Y. Chen, Y. Ge, R. Zhang, and Y. Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790, 2024 a

    2024 arXiv

  17. [26]

    K. Li, Z. Meng, H. Lin, Z. Luo, Y. Tian, J. Ma, Z. Huang, and T.-S. Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025

    2025 arXiv

  18. [27]

    L. Li, Y. Wei, Z. Xie, X. Yang, Y. Song, P. Wang, C. An, T. Liu, S. Li, B. Y. Lin, L. Kong, and Q. Liu. Vlrewardbench: A challenging benchmark for vision-language generative reward models. ArXiv, abs/2411.17451, 2024 b . URL https://api.semanticscholar.org/CorpusID:274281459

    2024 arXiv

  19. [28]

    H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  20. [29]

    J. Liu, Y. Song, B. Y. Lin, W. Lam, G. Neubig, Y. Li, and X. Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024 a

    2024 arXiv

  21. [30]

    Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer, 2024 b

    2024

  22. [31]

    Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X.-C. Yin, C.-L. Liu, L. Jin, and X. Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67 0 (12): 0 220102, 2024 c

    2024

  23. [32]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  24. [33]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME, 2024. URL https://maa.org/math-competitions/american-invitational-mathematics-examination-aime

    2024

  25. [34]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME, 2025. URL https://maa.org/math-competitions/american-invitational-mathematics-examination-aime

    2025

  26. [35]

    Mangalam, R

    K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36: 0 46212--46244, 2023

    2023

  27. [36]

    Masry, D

    A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  28. [37]

    Mathew, D

    M. Mathew, D. Karatzas, and C. Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209, 2021

    2021

  29. [38]

    Mathew, V

    M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697--1706, 2022

    2022

  30. [39]

    V. K. Nagaraja, V. I. Morariu, and L. S. Davis. Modeling context between objects for referring expression understanding. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pages 792--807. Springer, 2016

    2016

  31. [40]

    Computer-using agent: Introducing a universal interface for ai to interact with the digital world

    OpenAI. Computer-using agent: Introducing a universal interface for ai to interact with the digital world. 2025. URL https://openai.com/index/computer-using-agent

    2025

  32. [41]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructio...

    2022

  33. [42]

    Padlewski, M

    P. Padlewski, M. Bain, M. Henderson, Z. Zhu, N. Relan, H. Pham, D. Ong, K. Aleksiev, A. Ormazabal, S. Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. ArXiv preprint, abs/2405.02287, 2024

    2024 arXiv

  34. [43]

    Paiss, A

    R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3170--3180, 2023

    2023

  35. [44]

    R. Qiao, Q. Tan, G. Dong, M. Wu, C. Sun, X. Song, Z. GongQue, S. Lei, Z. Wei, M. Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    2024 arXiv

  36. [46]

    Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025 b

    2025 arXiv

  37. [47]

    Rahmanzadehgervi, L

    P. Rahmanzadehgervi, L. Bolton, M. R. Taesiri, and A. T. Nguyen. Vision language models are blind: Failing to translate detailed visual features into words, 2025. URL https://arxiv.org/abs/2407.06581

    2025 arXiv

  38. [48]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  39. [49]

    Rezatofighi, N

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658--666, 2019

    2019

  40. [50]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  41. [51]

    Sheng, C

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient rlhf framework. ArXiv preprint, abs/2409.19256, 2024. URL https://arxiv.org/abs/2409.19256

    2024 arXiv

  42. [52]

    C. Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  43. [53]

    P. Tong, E. Brown, P. Wu, S. Woo, A. J. V. IYER, S. C. Akula, S. Yang, J. Yang, M. Middepogu, Z. Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37: 0 87310--87356, 2024

    2024

  44. [54]

    K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37: 0 95095--95169, 2024 a

    2024

  45. [55]

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In A. Globersons, L. Mackey, D. Belgrav...

    2024

  46. [56]

    Y. Wang, B. Xu, Z. Yue, Z. Xiao, Z. Wang, L. Zhang, D. Yang, W. Wang, and Q. Jin. Timezero: Temporal video grounding with reasoning-guided lvlm. arXiv preprint arXiv:2503.13377, 2025

    2025 arXiv

  47. [57]

    Z. Wang, M. Xia, L. He, H. Chen, Y. Liu, R. Zhu, K. Liang, X. Wu, H. Liu, S. Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37: 0 113569--113697, 2024 c

    2024

  48. [58]

    Wu and S

    P. Wu and S. Xie. V*: Guided visual search as a core mechanism in multimodal llms, 2023. URL https://arxiv.org/abs/2312.14135

    2023 arXiv

  49. [59]

    Z. Wu, Z. Wu, F. Xu, Y. Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    2024 arXiv

  50. [60]

    Y. Xiao, E. Sun, T. Liu, and W. Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024

    2024 arXiv

  51. [61]

    L.-C.-T. Xiaomi. Mimo: Unlocking the reasoning potential of language model--from pretraining to posttraining. arXiv preprint arXiv:2505.07608, 2025

    2025 arXiv

  52. [62]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37: 0 52040--52094, 2024

    2024

  53. [63]

    T. Xie, J. Deng, X. Li, J. Yang, H. Wu, J. Chen, W. Hu, X. Wang, Y. Xu, Z. Wang, Y. Xu, J. Wang, D. Sahoo, T. Yu, and C. Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025. URL https://arxiv.org/abs/2505.13227

    2025

  54. [64]

    H. Xu, S. Xie, X. E. Tan, P.-Y. Huang, R. Howes, V. Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023

    2023 arXiv

  55. [65]

    Y. Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  56. [66]

    J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong. Dream 7b, 2025. URL https://hkunlp.github.io/blog/2025/dream

    2025

  57. [67]

    L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69--85. Springer, 2016

    2016

  58. [68]

    X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  59. [69]

    X. Yue, Y. Ni, T. Zheng, K. Zhang, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark...

    2024

  60. [70]

    X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang, H. Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024 c

    2024 arXiv

  61. [71]

    Zhang, B

    K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li, and Z. Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024 a . URL https://arxiv.org/abs/2407.12772

    2024 arXiv

  62. [72]

    Zhang, D

    R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, P. Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? ArXiv preprint, abs/2403.14624, 2024 b

    2024 arXiv

  63. [73]

    Zhang, H

    Y.-F. Zhang, H. Zhang, H. Tian, C. Fu, S. Zhang, J. Wu, F. Li, K. Wang, Q. Wen, Z. Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024 c

    2024 arXiv

  64. [74]

    Zheng, W

    L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023

    2023

  65. [75]

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  66. [76]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165--2183. PMLR, 2023

    2023

  67. [77]

    C. Zou, X. Guo, R. Yang, J. Zhang, B. Hu, and H. Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.