REVIEW 17 cited by
ChartBench: A Benchmark for Complex Visual Reasoning in Charts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in image understanding and generation. However, current benchmarks fail to accurately evaluate the chart comprehension of MLLMs due to limited chart types and inappropriate metrics. To address this, we propose ChartBench, a comprehensive benchmark designed to assess chart comprehension and data reliability through complex visual reasoning. ChartBench includes 42 categories, 66.6k charts, and 600k question-answer pairs. Notably, many charts lack data point annotations, which requires MLLMs to derive values similar to human understanding by leveraging inherent chart elements such as color, legends, and coordinate systems. We also design an enhanced evaluation metric, Acc+, to evaluate MLLMs without extensive manual or costly LLM-based evaluations. Furthermore, we propose two baselines based on the chain of thought and supervised fine-tuning to improve model performance on unannotated charts. Extensive experimental evaluations of 18 open-sourced and 3 proprietary MLLMs reveal their limitations in chart comprehension and offer valuable insights for further research. Code and dataset are publicly available at https://chartbench.github.io.
Forward citations
Cited by 17 Pith papers
-
ChartCap: Mitigating Hallucination of Dense Chart Captioning
A new 565K-pair chart-caption dataset with schema-based dense captions and a reference-free visual consistency metric improves VLM captioning and reduces hallucination.
-
ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts
A large-scale benchmark shows that leading multimodal language models still underperform expert humans at verifying climate claims from scientific charts.
-
LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
LongChart is a graph-consistent multi-chart VQA benchmark where 10 multimodal LLMs lose accuracy as question reasoning hops grow.
-
MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction
Current multimodal LLMs can copy the look of multi-view dashboards but mostly fail to bind real data and implement cross-view interactions.
-
Visual Programmability: A Guide for Code-as-Thought in Chart Understanding
A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.
-
FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models
A new benchmark of real-world financial charts shows current vision-language models lag badly on questions that require reading values from chart axes.
-
In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding
ChartScope, using a template-based synthetic data pipeline and dual-path reasoning training, outperforms prior chart-reading models on several advanced chart benchmarks.
-
UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis
UniGlyph replaces pre-rendered glyph conditions with segmentation-derived masks in a ControlNet diffusion model, reporting gains on visual text rendering benchmarks.
-
ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation
A fully automated pipeline generates a 222.5K-pair synthetic chart dataset with 27 chart types and 11 plotting libraries, and a GPT-4o-judged benchmark shows current open-weights VLMs still underperform on chart-to-co...
-
CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models
On six human-designed tests of reading charts and graphs, eight vision-language models scored below human participants and their error patterns fell far short of the human noise ceiling.
-
VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism
VReST combines Monte Carlo tree search with a self-reward signal inside a vision-language model to get higher accuracy than CoT, ToT, or voting baselines on MathVista, MathVision, and CharXiv, while spending several t...
-
Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
A new 1,430-item multimodal benchmark shows that leading multimodal LLMs rarely notice small visual traps needed for commonsense safety reasoning, with top scores near 0.46 on a scale whose maximum is about 0.97.
-
Data-to-Dashboard: Multi-Agent LLM Framework for Insightful Visualization in Enterprise Analytics
A multi-agent LLM system that detects the business domain of a raw dataset, generates domain-grounded insights, and renders them as charts, claims to beat single-prompt GPT-4o in insight quality.
-
SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model
SridBench provides a large multi-discipline benchmark for scientific illustration generation and shows current image generation models, especially GPT-4o-image, remain far below human expert quality.
-
Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant
The preprint's abstract claims a sparse softmax variant that masks non-competitive classes and accelerates training, but the provided body contains an unrelated chart-captioning paper and none of the claimed method.
-
Evaluating LLMs for Visualization Generation and Understanding
GPT-4o led on most chart generation and understanding tests in this sample, but all four models struggled with complex charts, dotted lines, and close bar lengths.
-
ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
ChartSketcher has a multimodal LLM sketch intermediate reasoning steps directly on chart images and feed those sketches back as visual feedback, improving chart QA accuracy over its base model.
Discussion (0). Sign in to comment.