REVIEW 33 cited by
ChartLlama: A Multimodal LLM for Chart Understanding and Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multi-modal large language models have demonstrated impressive performances on most vision-language tasks. However, the model generally lacks the understanding capabilities for specific domain data, particularly when it comes to interpreting chart figures. This is mainly due to the lack of relevant multi-modal instruction tuning datasets. In this article, we create a high-quality instruction-tuning dataset leveraging GPT-4. We develop a multi-step data generation process in which different steps are responsible for generating tabular data, creating chart figures, and designing instruction tuning data separately. Our method's flexibility enables us to generate diverse, high-quality instruction-tuning data consistently and efficiently while maintaining a low resource expenditure. Additionally, it allows us to incorporate a wider variety of chart and task types not yet featured in existing datasets. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset. ChartLlama outperforms all prior methods in ChartQA, Chart-to-text, and Chart-extraction evaluation benchmarks. Additionally, ChartLlama significantly improves upon the baseline in our specially compiled chart dataset, which includes new chart and task types. The results of ChartLlama confirm the value and huge potential of our proposed data generation method in enhancing chart comprehension.
Forward citations
Cited by 33 Pith papers
-
DataComp-VLM: Improved Open Datasets for Vision-Language Models
DataComp-VLM benchmark shows instruction-heavy data mixing outperforms filtering for VLM training, with DCVLM-Baseline achieving 63.6% on 33 tasks for 8B models (+5.4pp over FineVision).
-
ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats
ChartArena unifies eight chart families across three real-world visual scenarios and two languages under a format-agnostic triple/graph evaluation protocol, revealing clear gaps among 26 MLLMs.
-
ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats
ChartArena is a new benchmark dataset and evaluation protocol for chart parsing by MLLMs that covers numeric and diagrammatic charts in multiple languages and real-world visual conditions.
-
QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding
Introduces QCalEval benchmark showing best zero-shot VLM score of 72.3 on quantum calibration plots, with fine-tuning and in-context learning effects varying by model type.
-
SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
SciVisAgentBench provides 108 expert-crafted tasks and a mixed LLM-plus-deterministic evaluation pipeline for benchmarking AI agents that perform scientific visualization workflows.
-
SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
A taxonomy-grounded 108-case SciVis agent benchmark with multimodal LLM-plus-deterministic evaluation finds general-purpose coding agents currently beat most specialized SciVis agents on outcome quality.
-
InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information
InterChart is a new benchmark that reveals steep drops in VLM accuracy when moving from single-chart facts to integrative reasoning over 2-3 related charts, with better performance after decomposing complex charts.
-
LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
LongChart is a graph-consistent multi-chart VQA benchmark where 10 multimodal LLMs lose accuracy as question reasoning hops grow.
-
VisCanvas: A Node-based Interface for Exploratory Visualization Authoring with LLMs
A node-based interface for LLM chart authoring led 20 users to explore in more branched, tree-like patterns than a chat interface, without raising measured workload.
-
MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction
Current multimodal LLMs can copy the look of multi-view dashboards but mostly fail to bind real data and implement cross-view interactions.
-
Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
Introduces a benchmark for MLLM-based chart data extraction from unlabeled images and a human-centered training framework that reaches SOTA numerical accuracy with a 7B model.
-
DataComp-VLM: Improved Open Datasets for Vision-Language Models
DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.
-
TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL
TRON supplies 520 rule-verifiable online visual reasoning environments across five ability buckets that generate unlimited training instances for RL post-training, yielding consistent gains on ten external multimodal ...
-
ChartAct: A Benchmark for Dynamic Chart Understanding
ChartAct is a benchmark for dynamic chart understanding that shows most multimodal models struggle with interactive charts, with top performance at 84.5%.
-
ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
ChartFI-Bench supplies 896 chart-description pairs and four metrics (Faithfulness, Coverage, Informativeness, Acuity) to evaluate MLLM-generated chart descriptions on faithfulness and insightfulness.
-
ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
ChartFI-Bench supplies 896 chart-description pairs from visually complex charts and defines four metrics (Faithfulness, Coverage, Informativeness, Acuity) aligned to four quality dimensions to evaluate MLLM-generated ...
-
Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts
Chart-FR1 uses Focus-CoT for linking reasoning to visual cues and Focus-GRPO reinforcement learning with efficiency rewards to outperform prior MLLMs on dense chart reasoning tasks.
-
CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution
A 7B/8B model trained with decoupled tri-perspective SFT and QA-verified RL matches GPT-4o and approaches GPT-5 on chart-to-code generation benchmarks.
-
CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution
CharTide decouples chart-to-code data into three perspectives and uses inquiry-driven RL with atomic QA verification to let smaller VLMs surpass GPT-4o on chart-to-code tasks.
-
CharTool: Tool-Integrated Visual Reasoning for Chart Understanding
CharTool equips MLLMs with cropping and code tools plus agentic RL on DuoChart data to raise chart-reasoning accuracy by up to 9.78 percent on benchmarks.
-
ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch
ChartVerse uses Rollout Posterior Entropy and truth-anchored inverse QA synthesis to produce 640K high-quality chart reasoning samples, training an 8B model that surpasses its 30B teacher.
-
CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generation
CycleChart is a consistency-based framework that organizes chart generation, schema parsing, data parsing, and QA around single data instances to enforce bidirectional semantic alignment and improve cross-task generalization.
-
Visual Programmability: A Guide for Code-as-Thought in Chart Understanding
A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.
-
SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
SEAM measures VLM reasoning consistency across modalities using paired semantically equivalent textual and visual notations, and finds systematic vision-language imbalance.
-
BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning
Replotting real-world charts into code-backed images, then fine-tuning with supervised learning and GRPO reinforcement learning, produces chart QA models that beat prior chart-specific models on several benchmarks.
-
Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs
Aggregating a VLM's attention over visual tokens and mapping it back to image patches produces saliency maps that, in a 13-sample deletion test on ChartGemma, appear to localize the visual evidence behind chart answers.
-
ChatImage: Navigating Long-Form LLM Answers through Interactive Images
ChatImage renders LLM answers as images, then uses visual grounding to place clickable hotspots on rendered regions for interactive follow-up.
-
Demonstrating chart-plot: Closing the Last Mile of Academic Chart Generation
chart-plot is an agentic harness using style-aware code generation from venue figures, a LaTeX-aware render-and-revise loop, and structured edit handles to produce top-venue-ready academic charts.
-
From Data to Insights: Exploring Program-of-Thoughts Prompting for Chart Summarization
The work introduces a chart-to-dictionary auxiliary task paired with Program-of-Thoughts prompting to enable zero-shot chart summarization that matches existing methods on semantic and factual metrics.
-
VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
Merging a coding LLM into a vision-language model via task vectors yields an open-source multimodal coder that reaches near-GPT-4o performance on the authors' new benchmark.
-
AppAgent: Multimodal Agents as Smartphone Users
AppAgent lets large language models operate diverse smartphone apps via visual interactions and learns app usage from exploration or demonstrations.
-
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
A structured survey of multimodal code intelligence that formulates the field by code roles and organizes work into four domains while proposing verification-centered research directions.
-
Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction
Survey proposing a taxonomy for document parsing into pipeline-based systems and VLM-driven unified models, reviewing components, metrics, benchmarks, and challenges.
Discussion (0). Sign in to comment.