REVIEW 9 cited by
MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The mathematical capabilities of Multi-modal Large Language Models (MLLMs) remain under-explored with three areas to be improved: visual encoding of math diagrams, diagram-language alignment, and chain-of-thought (CoT) reasoning. This draws forth an urgent demand for an effective training paradigm and a large-scale, comprehensive dataset with detailed CoT rationales, which is challenging to collect and costly to annotate manually. To tackle this issue, we propose MAVIS, a MAthematical VISual instruction tuning pipeline for MLLMs, featuring an automatic data engine to efficiently create mathematical visual datasets. We design the data generation process to be entirely independent of human intervention or GPT API usage, while ensuring the diagram-caption correspondence, question-answer correctness, and CoT reasoning quality. With this approach, we curate two datasets, MAVIS-Caption (558K diagram-caption pairs) and MAVIS-Instruct (834K visual math problems with CoT rationales), and propose four progressive stages for training MLLMs from scratch. First, we utilize MAVIS-Caption to fine-tune a math-specific vision encoder (CLIP-Math) through contrastive learning, tailored for improved diagram visual encoding. Second, we also leverage MAVIS-Caption to align the CLIP-Math with a large language model (LLM) by a projection layer, enhancing vision-language alignment in mathematical domains. Third, we adopt MAVIS-Instruct to perform the instruction tuning for robust problem-solving skills, and term the resulting model as MAVIS-7B. Fourth, we apply Direct Preference Optimization (DPO) to enhance the CoT capabilities of our model, further refining its step-wise reasoning performance. Code and data will be released at https://github.com/ZrrSkywalker/MAVIS
Forward citations
Cited by 9 Pith papers
-
FormalAnalyticGeo: A Neural-Symbolic Based Framework for Multimodal Analytic Geometry Problem Generation
A neural-symbolic framework with a formal condition language and SDF rendering generates 7,043 multimodal analytic-geometry problems with visually measured ground truths at 0.70% median relative error.
-
Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation
A curated GPT-4o synthetic image dataset improves open-source generation models on instruction-following, surreal scenes, and multi-reference synthesis, plus two new benchmarks to measure those skills.
-
SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning
SyncLoop jointly evolves multimodal training data and model capability through alternating SFT and RL, selecting error-prone samples to improve geometry reasoning.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
An integrated survey organizing AI mathematical reasoning into informal, formal, discovery, and technique axes while cataloging benchmarks and assessing failure modes.
-
Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning
FiS-VLA embeds a diffusion-based action module into the final transformer blocks of a vision-language model, achieving 69% mean success on RLBench and a claimed 117.7 Hz control frequency.
-
Self-Reflective Reinforcement Learning for Diffusion-based Image Reasoning Generation
SRRL trains diffusion models to reason about images by repeatedly denoising, evaluating, and re-noising their own samples across several reflective rounds.
-
Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start
Using a two-stage SFT-plus-GRPO pipeline, the authors train 3B and 7B multimodal models that beat prior open-source models on four math reasoning benchmarks, while showing that reflective 'aha moment' text is not a re...
-
Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
In autoregressive image generation, DPO is stronger on in-domain prompts, GRPO generalizes better out-of-domain, and reward model generalization partly shapes both.
Discussion (0). Sign in to comment.