Pith. sign in

REVIEW 10 cited by

M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.04387 v2 pith:JMQ2GASY submitted 2023-06-07 cs.CV cs.CL

Lei Li , Yuwei Yin , Shicheng Li , Liang Chen , Peiyi Wang , Shuhuai Ren , Mukai Li , Yazheng Yang
show 4 more authors
Jingjing Xu Xu Sun Lingpeng Kong Qi Liu
This is my paper
classification cs.CVcs.CL
keywords datasetinstructioninstructionsdatasetstaskstuningadvancedhuman
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Instruction tuning has significantly advanced large language models (LLMs) such as ChatGPT, enabling them to align with human instructions across diverse tasks. However, progress in open vision-language models (VLMs) has been limited due to the scarcity of high-quality instruction datasets. To tackle this challenge and promote research in the vision-language field, we introduce the Multi-Modal, Multilingual Instruction Tuning (M$^3$IT) dataset, designed to optimize VLM alignment with human instructions. Our M$^3$IT dataset comprises 40 carefully curated datasets, including 2.4 million instances and 400 manually written task instructions, reformatted into a vision-to-text structure. Key tasks are translated into 80 languages with an advanced translation system, ensuring broader accessibility. M$^3$IT surpasses previous datasets regarding task coverage, instruction number and instance scale. Moreover, we develop Ying-VLM, a VLM model trained on our M$^3$IT dataset, showcasing its potential to answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese. We have open-sourced the dataset to encourage further research.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types

    cs.CV 2025-02 conditional novelty 7.0 of 10

    The authors release TaskGalaxy, a 413k-sample multimodal instruction dataset with 19,227 hierarchically organized task types, and report consistent benchmark gains when it is added to LLaVA-v1.5 and InternVL-Chat-v1.0.

  2. LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    LLaSO releases a 3.8B speech-language model, 25.5M training instances, and an evaluation benchmark, claiming a normalized score of 0.72.

  3. Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security

    cs.CR 2025-07 conditional novelty 6.0 of 10

    An iterative attacker-defender reinforcement learning method that makes a multimodal LLM refuse more jailbreak prompts without over-refusing ordinary queries.

  4. VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models

    eess.IV 2025-07 conditional novelty 6.0 of 10

    A perturbation computed from early attention and value matrices can make LLaVA, Instruct-BLIP, and BLIP2-T5 fail to detect objects inside a specified image region while keeping the rest of the image usable.

  5. M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset

    cs.CL 2025-06 conditional novelty 6.0 of 10

    M3FinMeeting is a new 600-meeting, trilingual, multi-sector benchmark with three financial meeting understanding tasks, on which current LLMs achieve only moderate judged quality scores.

  6. Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning

    cs.CV 2026-03 conditional novelty 5.5 of 10

    Goal-driven selection of 1× multimodal instruction subsets reaches a 512k Uni-10x baseline after ~27–35k samples and improves accuracy by up to +3.08 pp under a fixed Qwen3-VL recipe.

  7. Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Selective fine-tuning of language-specific shallow layers identified by neuron activation analysis improves multilingual vision-language performance with only 14% of parameters tuned.

  8. Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Instructify converts image metadata into visual instruction-tuning conversations with open LLMs, matching or exceeding GPT-4-generated data quality on LMM benchmarks.

  9. A Survey on Video Temporal Grounding with Multimodal Large Language Model

    cs.CV 2025-08 unverdicted novelty 3.0 of 10

    A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.

  10. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools