The reviewed record of science sign in
Pith

arxiv: 2410.18558 · v2 · pith:NKVEBGQI · submitted 2024-10-24 · cs.CL

Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data

Shuhao Gu , Jialing Zhang , Siyuan Zhou , Kevin Yu , Zhaohu Xing , Liangdong Wang , Zhou Cao , Jintao Jia
show 18 more authors
Zhuoyi Zhang Yixuan Wang Zhenchong Hu Bo-Wen Zhang Jijie Li Dong Liang Yingli Zhao Songjing Wang Yulong Ao Yiming Ju Huanhuan Ma Xiaotong Li Haiwen Diao Yufeng Cui Xinlong Wang Yaoqi Liu Fangxiang Feng Guang Liu
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:NKVEBGQIrecord.jsonopen to challenge →

classification cs.CL
keywords datainstructionmultimodaldatasetshigh-qualityinfinity-mmlarge-scalemodels
0
0 comments X
read the original abstract

Recently, Vision-Language Models (VLMs) have achieved remarkable progress in multimodal tasks, and multimodal instruction data serves as the foundation for enhancing VLM capabilities. Despite the availability of several open-source multimodal datasets, limitations in the scale and quality of open-source instruction data hinder the performance of VLMs trained on these datasets, leading to a significant gap compared to models trained on closed-source data. To address this challenge, we introduce Infinity-MM, a large-scale multimodal instruction dataset. We collected the available multimodal instruction datasets and performed unified preprocessing, resulting in a dataset with over 40 million samples that ensures diversity and accuracy. Furthermore, to enable large-scale expansion of instruction data and support the continuous acquisition of high-quality data, we propose a synthetic instruction generation method based on a tagging system and open-source VLMs. By establishing correspondences between different types of images and associated instruction types, this method can provide essential guidance during data synthesis. Leveraging this high-quality data, we have trained a 2-billion-parameter Vision-Language Model, Aquila-VL-2B, which achieves state-of-the-art (SOTA) performance among models of similar scale. The data is available at: https://huggingface.co/datasets/BAAI/Infinity-MM.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Emu3.5: Native Multimodal Models are World Learners

    cs.CV 2025-10 unverdicted novelty 6.0

    Emu3.5 is a native multimodal world model pre-trained on over 10 trillion vision-language tokens with next-token prediction, post-trained via reinforcement learning, and accelerated by Discrete Diffusion Adaptation fo...

  2. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    cs.CV 2025-08 unverdicted novelty 6.0

    InternVL3.5 advances open-source multimodal models with Cascade RL for +16% reasoning gains and ViR for 4x inference speedup, with the 241B model reaching SOTA among open-source MLLMs on multimodal, reasoning, and age...

  3. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

    cs.CV 2025-04 conditional novelty 6.0

    InternVL3-78B sets a new open-source SOTA of 72.2 on MMMU via native joint multimodal pre-training, V2PE, MPO, and test-time scaling while remaining competitive with proprietary models.

  4. Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos

    cs.CV 2025-01 conditional novelty 6.0

    Sa2VA unifies SAM-2 segmentation with MLLM reasoning into a single model for referring segmentation and conversation on images and videos, supported by a new 72k-expression Ref-SAV dataset.

  5. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

    cs.CV 2024-12 unverdicted novelty 6.0

    InternVL 2.5 is the first open-source MLLM to surpass 70% on the MMMU benchmark via model, data, and test-time scaling, with a 3.7-point gain from chain-of-thought reasoning.

  6. Bridging Video Understanding and Generation in a Unified Framework

    cs.CV 2026-06 unverdicted novelty 5.0

    Vega unifies video understanding and generation via shared vocabulary and hybrid autoregressive-diffusion architecture, reporting strong results on VBench and VideoMME.

  7. SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning

    cs.CV 2026-06 unverdicted novelty 5.0

    SingGuard presents a policy-adaptive multimodal LLM guardrail family with hybrid reasoning regimes and a new benchmark of 56,340 examples, claiming SOTA F1 across 35 datasets and improved policy adherence under runtim...

  8. SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning

    cs.CV 2026-06 unverdicted novelty 5.0

    SingGuard introduces a policy-adaptive multimodal LLM guardrail with dynamic reasoning regimes and SingGuard-Bench, reporting SOTA F1 scores across 35 datasets and improved policy-following accuracy under runtime shifts.

  9. Kwai Keye-VL-2.0 Technical Report

    cs.CV 2026-06 unverdicted novelty 4.0

    Kwai Keye-VL-2.0-30B-A3B is a 30B MoE model with 3B active parameters using DSA adaptation and MOPD distillation that reports SOTA results on video understanding and agent benchmarks.