SVIT: Scaling up Visual Instruction Tuning

Boya Wu; Bo Zhao; Muyang He; Tiejun Huang

SVIT: Scaling up Visual Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.04087 v3 pith:S6KUP7KD submitted 2023-07-09 cs.CV

Bo Zhao , Boya Wu , Muyang He , Tiejun Huang This is my paper

classification cs.CV

keywords visualdatainstructionmodelsmultimodaltuningdatasetpairs

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Improving Large Vision and Language Models by Learning from a Panel of Peers
cs.CV 2025-09 conditional novelty 6.0 of 10

A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.
Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models
cs.CV 2025-07 conditional novelty 5.5 of 10

A monolithic multimodal LLM that cuts pre-training data by 58% and first-token latency by up to 69% while matching or beating its predecessor on 15 benchmarks.
StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design
cs.CV 2026-07 conditional novelty 5.0 of 10

A co-designed 0.9B UI vision-language model tops small-model benchmarks on screen Q&A and Chinese OCR and runs on a Snapdragon 8 Gen5 at 98 tokens/second.
Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
cs.CV 2025-08 conditional novelty 5.0 of 10

MTMEUR is a new multimodal emotion reasoning benchmark where the best single model scores 71.19% and a four-agent reasoning framework tops 72.93%.
GLAD: Generalizable Tuning for Vision-Language Models
cs.CV 2025-07 conditional novelty 5.0 of 10

GLAD improves few-shot CLIP generalization by fusing original and sharpness-aware gradients during LoRA tuning and adding image-conditional text alignment.
SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
cs.CL 2025-06 conditional novelty 5.0 of 10

SMAR, a KL-based regularizer on per-modality expert routing, retains 86.6% of a Mixtral 8x7B's language score during visual instruction tuning with only 2.5% pure-text data.

Pith tools