REVIEW 5 cited by
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.
Forward citations
Cited by 5 Pith papers
-
ChartCap: Mitigating Hallucination of Dense Chart Captioning
A new 565K-pair chart-caption dataset with schema-based dense captions and a reference-free visual consistency metric improves VLM captioning and reduces hallucination.
-
ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
ChartMind is a new bilingual chart QA benchmark, and ChartLLM's structured context extraction yields higher scores than three existing prompting paradigms in the paper's evaluations.
-
CHAOS: Chart Analysis with Outlier Samples
A chart perturbation robustness benchmark with five textual and ten visual distortion types, three human-calibrated severity levels, and evaluations of 13 MLLMs on ChartQA and chart summarization.
-
Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant
The preprint's abstract claims a sparse softmax variant that masks non-competitive classes and accelerates training, but the provided body contains an unrelated chart-captioning paper and none of the claimed method.
-
ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
ChartSketcher has a multimodal LLM sketch intermediate reasoning steps directly on chart images and feed those sketches back as visual feedback, improving chart QA accuracy over its base model.
Discussion (0). Continue with ORCID to comment.