Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multimodal model that draws on the chart as it reasons, then reads its own drawings back, beats its base model and, on most chart benchmarks, GPT-4o.

desk verdict Solid engineering and plausible gains, but the oracle-annotation cold start and unresolved EvoChart contamination need to be settled before the visual-feedback story carries. read the letter →

arxiv 2505.19076 v1 pith:QX4UWYFM submitted 2025-05-25 cs.CV

Muye Huang , Lingling Zhang , Jie Ma , Han Lai , Fangzhi Xu , Yifei Li , Wenjun Wu , Yaqiang Wu
show 1 more author
Jun Liu
This is my paper · ORCID
classification cs.CV
keywords chartunderstandingmultimodallargelanguagemodelssketch-basedchain-of-thoughtvisualfeedbackself-reflectionreinforcementlearningMonteCarlotreesearchquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a chart-reading multimodal model reasons more accurately when it is allowed to sketch on the image while it thinks: it marks points, draws guide lines, and then sees its own annotations as the next input. This Sketch-CoT loop is combined with a reflection step in which the model notices when its marks are misplaced and redraws them, mimicking a human correcting a sketch. The authors construct 300K training traces by distilling reasoning chains from a text-only LLM supplied with ground-truth chart annotations, with half the traces containing deliberately injected coordinate errors for the model to fix, then refine the model with off-policy reinforcement learning over sketches produced by a Monte Carlo tree search. On six chart benchmarks, the resulting 72B model surpasses its Qwen2-VL base and, on five of the six, also beats GPT-4o; even the 2B version improves over its base. The paper's central claim is that the visual feedback loop and the reflection stage carry the gain, not scale or extra data alone.

What carries the argument

The load-bearing mechanism is the Sketch-CoT loop: a programmatic sketching library expressed as pseudocode (BEGIN...END blocks, normalized coordinates on a unit square, commands such as create_point, create_line, create_circle, translate, rotate, and delete) that the model can invoke anywhere in its reasoning, paired with a rendering pipeline that turns the emitted code into an overlay on the chart and feeds the rendered image back into the multimodal dialogue. Reflection is trained in by mixing traces with deliberately wrong coordinates that the model then corrects, and the sketch-MCTS variant of Monte Carlo tree search supplies step-level correct and incorrect trajectories, with rewards from a separate value model and duplicate or subset drawing-code nodes pruned, for the off-policy RL stage. The loop ends when the model stops emitting drawing code, at which point it gives its final answer.

What would settle it

Take a trained ChartSketcher model, run it on chart questions, and silently corrupt a fraction of the rendered sketch overlays before they are fed back, for instance translating every drawn point by a fixed offset. If final accuracy is unchanged, or if the model's subsequent drawing code is identical to the uncorrupted run, then the visual feedback is not carrying the reasoning load and the reflection stage is imitating scripted corrections rather than rereading its own drawings.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that a multimodal large language model can ground its chain of thought in the image by drawing: at any intermediate step it may emit commands in a small programmatic sketching language, a renderer draws those commands onto the chart, and the updated image is fed back into the dialogue as the next visual input. Because the model literally sees the effect of each reasoning step, it can notice when a point lands on the wrong bar or the wrong axis position and issue corrected drawing code. The paper reports that this mechanism, trained through a cold-start distillation phase (300K samples, half with scripted reflection) followed by a KTO-based reinforcement-learning phase over sketch-MCTS search trees (50K samples), lifts ChartSketcher-72B to 85.20 on ChartQA-H, 92.64 on ChartQA-A, 63.28 on EvoChart-QA, 68.33 on ChartBench, 76.72 on PlotQA, and 76.59 on VisualCoT, above Qwen2VL-72B everywhere and above GPT-4o on five of the six chart benchmarks.

Load-bearing premise

The training traces instruct the model to draw at coordinates taken from ground-truth chart annotations, so the cold-start model has never practiced converting pixels into coordinates; the whole improvement depends on this transferring to test-time charts, where the model must localize elements from the image alone, and on the feedback loop correcting rather than amplifying its own errors.

Editorial extensions

If this is right

  • ChartSketcher-72B beats Qwen2VL-72B on every chart benchmark in Table 1 and beats GPT-4o on five of the six, so the paper attributes the gain to the sketch-feedback method rather than to parameter scale.
  • The retention of general vision ability (OpenImages 68.68 vs. 51.75 for the base; VisualCoT composite 76.59 vs. 72.14) implies the mechanism transfers beyond charts to natural images.
  • Replacing the RL stage with SFT on identical data drops the VisualCoT composite to 70.95, below the base model, so the off-policy search-tree sampling is what endows the generalization behavior.
  • Removing the multimodal feedback (w/o Feedback) hurts hardest on the most visually demanding chart tasks (EvoChart-QA falls from 63.28 to 57.76), indicating the feedback loop matters precisely when pixel-level extraction is hard.
  • The final annotated chart, including visible corrections, gives users an inspectable artifact of how the answer was reached.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same draw-feedback loop should transfer to other visually dense reasoning domains where localization and reading interact, such as documents, maps, GUI screenshots, and medical images, since the paper's own general-vision results suggest the mechanism is generic, but those targets are never tested here.
  • Editorial inference: the cold-start dependency on oracle annotations creates a bootstrapping opportunity: once the RL-trained model can produce reliable sketches on unannotated charts, it could generate its own sketch traces with self-checked coordinates, removing the annotation requirement entirely.
  • Editorial inference: because each sketch step renders and re-encodes an image, inference cost grows with step count, and the paper caps the chain at 12; a cost-adjusted comparison against text-only CoT would reveal whether visual grounding is worth its overhead.
  • Editorial inference: a direct test of whether reflection is genuine rather than scripted would be to corrupt the rendered overlay mid-reasoning and check whether the model's next drawing code changes in response; the paper does not include such a probe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ChartSketcher, a chart understanding model that augments a Qwen2VL backbone with a programmatic sketching library and a multi-turn visual feedback loop. The training pipeline has a cold-start stage that distills sketch-based chain-of-thought reasoning from an LLM given oracle chart annotations, followed by an off-policy reinforcement learning stage (Sketch-MCTS with KTO) on partially unlabeled data. The authors report gains over Qwen2VL-72B and several proprietary models on ChartQA, EvoChart-QA, ChartBench, PlotQA, and VisualCoT, and attribute these improvements to Sketch-CoT and the reflection loop. The paper includes ablations and an analysis of reasoning length and rethink frequency.

Significance. If the results hold, ChartSketcher is a valuable contribution to multimodal reasoning, introducing an interpretable, interactive sketching mechanism that differs from text-only CoT and cropping-based methods. The promise of released code and data is a strength. However, the current evidence for the central mechanism is weakened by potential training/evaluation overlap, a mismatch between oracle-supervised cold-start training and pixel-based inference, and a feedback-removal ablation that is confounded with input distribution shift. The core idea deserves scrutiny, but the experimental support as presented is not yet convincing.

major comments (4)
  1. [Section 4.1, Table 1] The RL training data is described as "including ChartQA, ChartBench, and VisualCoT," and the cold-start data uses images from EvoChart-Corpus and seed questions from EvoChart-QA. The evaluation in Table 1 reports results on exactly these benchmarks. The paper does not state which splits are used for training and evaluation. If the test splits of ChartQA, ChartBench, VisualCoT, or EvoChart-QA overlap with the training data, the reported improvements on these benchmarks are not evidence of generalization. Please specify the exact splits, or remove all benchmarks that share data with the training set.
  2. [Section 3.2.1] The cold-start traces are distilled from an LLM that is given the ground-truth annotations of EvoChart-Corpus, and the distillation prompt forbids revealing the annotations. Every training trace therefore contains oracle drawing coordinates. At inference the model must produce these coordinates from pixels alone. The paper provides no measurement of the spatial accuracy of the model's self-generated sketches, and the Sketch-MCTS reward (Algorithm 1, line 15) is answer correctness, not coordinate accuracy. This makes it possible that the model learns a text-correction template rather than a visual-grounding skill, and that the feedback loop amplifies localization errors. Please report a quantitative comparison of self-generated sketch coordinates with ground-truth coordinates on a held-out set, and/or add an ablation in which the oracle coordinates are corrupted during cold-start training.
  3. [Section 4.1 (Evaluation Metrics)] All reported numbers are produced by a DeepSeek-Distill-Qwen-32B judge with a 3-run majority vote. No human agreement study, confidence intervals, or significance tests are given. Claims such as "significant improvements" in Section 4.2 are not supported by statistical evidence. Please report judge-human agreement and error bars (e.g., bootstrapped over evaluation questions), or at least acknowledge the uncertainty.
  4. [Section 4.3 (w/o Feedback)] The w/o Feedback ablation replaces the rendered feedback image with an empty string. Since every training example includes the rendered image, this condition simultaneously removes the visual feedback and changes the input distribution. The observed drop on EvoChart and ChartQA-H is therefore not a clean measure of the causal role of visual feedback. An additional control that supplies a fixed, non-informative sketch in the same image slot (or a model trained with randomly dropped feedback images) would isolate the feedback mechanism.
minor comments (4)
  1. [Table 1 vs Table 2] There is an inconsistency for Claude-3.5: Table 1 lists ChartBench=56.96 and PlotQA=57.63, whereas Table 2(a) lists ChartBench=57.63 and PlotQA=60.64. The numbers appear swapped, suggesting a copy-paste error. Please correct the tables and re-verify all reported values.
  2. [Section 4.3 (w/o CoT)] The w/o CoT setting is described as "does not apply the ChartSketcher methodology," but it is unclear whether the model is trained on the same data with the sketch code removed or simply fine-tuned on the original Qwen2VL output. Please clarify what data and loss are used for this baseline.
  3. [Figure 3] Figure 3 is difficult to read: the bars are not labeled for "Avg CoT Right/Wrong" and "Avg Rethink Right/Wrong," and the y-axes are not shown. Please add axis labels and a legend, and state the units clearly.
  4. [Notation] The abbreviation "VisCoT" in Table 2 should be consistent with "VisualCoT" used elsewhere in the paper.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline gains on EvoChart, ChartBench and VisualCoT are not independent: the paper trains on these benchmarks and then reports them as held-out predictions, without documenting any split.

  1. fitted input called prediction [Section 3.2.1 (Sketch-CoT Data Synthesis) and Section 4.1 (Data Construction), with results in Table 1]
    "We used the EvoChart-Corpus [13], a dataset with high-quality synthesized chart images... we used all the images and part of the QA pairs from EvoChart-Corpus. ... During the cold start phase, our base dataset images were sourced from EvoChart-Corpus, with seed questions from ChartQA [27] and EvoChart-QA [13]."

    EvoChart-Corpus and EvoChart-QA are the same authors' benchmark (ref [13]). The training set is built from the corpus images and QA pairs, with EvoChart-QA used as a seed question source, while Table 1 reports ChartSketcher's performance on EvoChart-QA as a headline result. No train/test split is specified, so the 63.28 EvoChart-QA score is not a held-out prediction; the model was fitted on data derived from the same benchmark and then this benchmark is presented as evidence of generalization.

  2. fitted input called prediction [Section 4.1 (Data Construction) and Table 1]
    "For the RL phase, we conducted training across multiple datasets, including ChartQA, ChartBench [51], and VisualCoT."

    The RL phase trains on ChartBench and VisualCoT, and the same Table 1 evaluates ChartSketcher on ChartBench (68.33) and VisualCoT (76.59). Without a documented separation of train and test instances within these benchmarks, the reported gains over Qwen2VL-72B cannot be attributed to the Sketch-CoT/reflection mechanism rather than to the model having been trained on the benchmark itself. The 'prediction' is thus at least partly fitted input renamed as a result.

full rationale

The paper's central mechanism (Sketch-CoT with visual feedback and reflection) is not itself definitionally circular: the model is trained to emit drawing code and read back rendered images, and the ablation 'w/o Feedback' in Table 1 is a legitimate (if imperfect) test of the feedback channel. The elevated circularity score comes from the evaluation design, not from the method's internal logic. Two load-bearing reported numbers are not independent: (1) EvoChart-QA is a self-authored benchmark whose corpus and seed questions are used in cold-start training, and (2) ChartBench and VisualCoT are explicitly included in the RL training mix and then reported as evaluation benchmarks. The paper nowhere states that held-out splits were used, so these results are consistent with the model having memorized or been optimized toward the test distribution. The cold-start oracle-annotation transfer issue and the DeepSeek judge used for both reward and evaluation are additional threats to clean attribution, but they are correctness concerns rather than reductions by construction; the two train/test overlaps are sufficient to place this at the boundary of partial circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The pipeline rests on oracle-derived synthetic data, LLM-based reward and grading, and hand-picked MCTS and training hyperparameters. No new physical or conceptual entities are introduced.

free parameters (2)
  • MCTS termination and sampling hyperparameters = SIM_lim=25, SUCC_lim=3, MAX_DEPTH=8, Cmax=6, MAX_CHILD=3, CPUCT=3.0, Thigh=0.9, Tlow=0.4
    Chosen by hand in Section 4.1 and Algorithm 1 with no sensitivity analysis. These control the RL data quality and directly affect the claimed performance.
  • Training hyperparameters = LoRA rank 16, alpha 32, lr 1e-4 cold and 1e-5 RL, 4 cold epochs, 1 RPO epoch, 1 KTO epoch
    Reported in Training Settings as fixed choices without ablations or grid search. They are necessary to reproduce the reported results.
assumptions (4)
  • domain assumption EvoChart-Corpus annotations are accurate and representative of real-world charts.
    All cold-start Sketch-CoT data is distilled from these annotations (Section 3.2.1), so errors or biases in the corpus propagate into the training data.
  • domain assumption DeepSeek-Distill-Qwen-32B voting reliably measures answer correctness against ground truth.
    Evaluation metrics in Section 4.1 rely entirely on this judge, with no human agreement or calibration check reported.
  • domain assumption DeepSeek-Distill-Qwen-14B value network provides reliable rewards in sketch-MCTS.
    The RL phase in Section 3.2.2 depends on this network to label positive and negative nodes, so reward noise directly affects what the model learns.
  • domain assumption MLLMs can learn to interpret serialized visual feedback through cold-start training.
    The architecture in Section 3.1 assumes the model can read multi-turn rendered sketch images; the paper does not measure feedback comprehension independently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding." pith.science (2026). https://pith.science/paper/QX4UWYFM

@misc{pith2026250519076,
  author       = {Pith},
  title        = {Pith review of: ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QX4UWYFM}},
  note         = {Machine review of arXiv:2505.19076}
}
read the original abstract

Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understanding poses significant challenges to existing multimodal large language models (MLLMs) due to the need for precise and complex visual reasoning. Current step-by-step reasoning models primarily focus on text-based logical reasoning for chart understanding. However, they struggle to refine or correct their reasoning when errors stem from flawed visual understanding, as they lack the ability to leverage multimodal interaction for deeper comprehension. Inspired by human cognitive behavior, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method designed to address these limitations. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reasoning process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. We employ a two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning to enhance reflection and generalization. Experiments demonstrate that ChartSketcher achieves promising performance on chart understanding benchmarks and general vision tasks, providing an interactive and interpretable approach to chart comprehension.

Figures

Figures reproduced from arXiv: 2505.19076 by the authors.

Figure 1
Figure 1. The overview of the proposed ChartSketcher. Dashed lines indicate intermediate reasoning [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of ChartSketcher Training Process. The upper part illustrates the cold start [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Analysis of CoT length and the number of rethink iterations for both correctly and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Four cases for ChartSketcher. The drawing code associated with each step is omitted for [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: An illustrative example of a challenge in current MLLMs [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The synthesis process of Sketch-CoT. The left side illustrates the CoT synthesis process [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Visualization of an additional case study on PlotQA. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Visualization of an additional case study on ChartQA. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Visualization of an additional case study on general image. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Visual Programmability: A Guide for Code-as-Thought in Chart Understanding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.

Reference graph

Works this paper leans on

65 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    2023

  2. [2]

    Onechart: Purify the chart structural extraction via one auxiliary token

    Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Onechart: Purify the chart structural extraction via one auxiliary token. arXiv preprint arXiv:2404.09987, 2024

    arXiv 2024

  3. [3]

    M 3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand...

    work page 2024

  4. [4]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, and Zhaoyang Liu. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    work page 2025

  5. [5]

    Vision-language models can self-improve reasoning via reflection, 2024

    Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. Vision-language models can self-improve reasoning via reflection, 2024

    work page 2024

  6. [6]

    Caparena: Benchmarking and analyzing detailed image captioning in the llm era, 2025

    Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. Caparena: Benchmarking and analyzing detailed image captioning in the llm era, 2025

    work page 2025

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    work page 2025

  8. [8]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. CoRR, abs/2411.14432, 2024. 10

    arXiv 2024

Show all 65 references
  1. [9]

    KTO: model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR, abs/2402.01306, 2024

    2024 arXiv

  2. [10]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024

    2024

  3. [11]

    Chartllama: A multimodal LLM for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal LLM for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023

    2023 arXiv

  4. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Op...

    2022

  5. [13]

    Evochart: A benchmark and a self-training approach towards real-world chart understanding

    Muye Huang, Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu. Evochart: A benchmark and a self-training approach towards real-world chart understanding. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advan...

    2025

  6. [14]

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V . Jawahar. ICDAR2019 competition on scanned receipt OCR and information extraction. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, Se...

    2019

  7. [15]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700–6709. Computer Vision...

    2019

  8. [16]

    Figureqa: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. In ICLR, 2018

    2018

  9. [17]

    Álvarez, Adrià Recasens, and Àgata Lapedriza

    Ronak Kosti, José M. Álvarez, Adrià Recasens, and Àgata Lapedriza. Context based emotion recognition using EMOTIC dataset. IEEE Trans. Pattern Anal. Mach. Intell., 42(11):2755–2766, 2020

    2020

  10. [18]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship ...

    2018 arXiv

  11. [19]

    Blaschko, Lukasz Borchmann, Mickaël Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel Józiak, and Ernest Valveny

    Jordy Van Landeghem, Rafal Powalski, Rubèn Tito, Dawid Jurkiewicz, Matthew B. Blaschko, Lukasz Borchmann, Mickaël Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel Józiak, and Ernest Valveny. Document understanding dataset and evaluation (DU...

    2023

  12. [20]

    Pix2struct: Screenshot parsing as pretraining for visual language understanding

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In ICML, pages 202: 18893–18...

    2023

  13. [21]

    Classification-regression for chart comprehension

    Matan Levy, Rami Ben-Ari, and Dani Lischinski. Classification-regression for chart comprehension. In ECCV, pages 13696: 469–484, 2022

    2022

  14. [22]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  15. [23]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Se...

    2024

  16. [24]

    Matcha: Enhancing visual language pretraining with math reasoning and chart derendering

    Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. In ACL, pages 12756–12770, 2023

    2023

  17. [25]

    Chain-of-spot: Interactive reasoning improves large vision-language models

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. CoRR, abs/2403.12966, 2024. 11

    2024 arXiv

  18. [26]

    Unichart: A universal vision-language pretrained model for chart comprehension and reasoning

    Ahmed Masry, Parsa Kavehzadeh, Do Xuan Long, Enamul Hoque, and Shafiq Joty. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. In EMNLP, pages 14662–14684, 2023

    2023

  19. [27]

    Joty, and Enamul Hoque

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, pages 2263–2279, 2022

    2022

  20. [28]

    Rizwan Parvez, Enamul Hoque, and Shafiq Joty

    Ahmed Masry, Mehrad Shahmohammadi, Md. Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartin- struct: Instruction tuning for chart comprehension and reasoning. arXiv preprint arXiv:2403.09028 , 2024

    2024 arXiv

  21. [29]

    Chart- gemma: Visual instruction-tuning for chart reasoning in the wild, 2024

    Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chart- gemma: Visual instruction-tuning for chart reasoning in the wild, 2024

    2024

  22. [30]

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V . Jawahar. Infographicvqa. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pages 2582–2591. IEEE, 2022

    2022

  23. [31]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 2199–2208. IEEE, 2021

    2021

  24. [32]

    Char- tassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning

    Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Char- tassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv: 2401.02384, 2024

    2024 arXiv

  25. [33]

    Khapra, and Pratyush Kumar

    Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In WACV, pages 1516–1525, 2020

    2020

  26. [34]

    Openai o1 system card, 2024

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, and Alex Tachard Passos Zhuohan Li. Openai o1 system card, 2024

    2024

  27. [35]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, and Florencia Leoni Aleman et al. Gpt-4 technical report, 2024

    2024

  28. [36]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision, ICCV 2015...

    2015

  29. [37]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, and Zihan Qiu. Qwen2.5 technical report, 2025

    2025

  30. [38]

    Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In Amir Globersons, Lester Mackey, Danielle Belgrav...

    2024

  31. [39]

    Textcaps: A dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: A dataset for image captioning with reading comprehension. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasg...

    2020

  32. [40]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8317–8326...

    2019

  33. [41]

    Dai, and Anja Hauth

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, and Anja Hauth. Gemini: A family of highly capable multimodal models, 2024. 12

    2024

  34. [42]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and Soroosh Mariooryad et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    2024

  35. [43]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with llms, 2025

    2025

  36. [44]

    Qvq: To see the world with wisdom, December 2024

    Qwen Team. Qvq: To see the world with wisdom, December 2024

    2024

  37. [45]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  38. [46]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011

    2011

  39. [47]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024

  40. [48]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,...

    2022

  41. [49]

    Interactive evolution: A neural-symbolic self-training framework for large language models, 2024

    Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. Interactive evolution: A neural-symbolic self-training framework for large language models, 2024

    2024

  42. [50]

    Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning, 2025

    Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, and Zhiyong Wu. Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning, 2025

    2025

  43. [51]

    Chartbench: A benchmark for complex visual reasoning in charts

    Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for complex visual reasoning in charts. CoRR, abs/2312.15915, 2023

    2023 arXiv

  44. [52]

    Chartmoe: Mixture of diversely aligned expert connector for chart understanding, 2025

    Zhengzhuo Xu, Bowen Qu, Yiyan Qi, Sinan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chartmoe: Mixture of diversely aligned expert connector for chart understanding, 2025

    2025

  45. [53]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, and Fei Huang. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  46. [54]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Adva...

    2023

  47. [55]

    Relative preference optimization: Enhancing LLM alignment through contrasting responses across identical and diverse prompts

    Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing LLM alignment through contrasting responses across identical and diverse prompts. CoRR, abs/2402.10958, 2024

    2024 arXiv

  48. [56]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014

    2014

  49. [57]

    Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning

    Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning. arXiv preprint arXiv: 2404.16635, 2024

    2024 arXiv

  50. [58]

    Fung, Christopher Thomas, Heng Ji, and Shih-Fu Chang

    Mingyang Zhou, Long Chen Yi R. Fung, Christopher Thomas, Heng Ji, and Shih-Fu Chang. Enhanced chart understanding in vision and language task via cross-modal pre-training on plot table pairs. InFindings of ACL, 2023

    2023

  51. [59]

    good" in the year 2015 based on the provided graph. First, I need to understand what the graph is showing. The title says

    Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pages 4995–5004. IEEE Computer Society, 2016. 13...

    2016

  52. [60]

    For example: create_point p1 0.2 0.2 red

    How can I change the color of a shape? Specify the color in the creation command. For example: create_point p1 0.2 0.2 red

  53. [61]

    Commands after END are ignored

    What does END do? The END command stops the pseudocode execution. Commands after END are ignored

  54. [62]

    Always include BEGIN and END

    What happens if I forget END? If END is missing, the parser continues parsing until the last line. Always include BEGIN and END

  55. [63]

    - Rotation: Rotates the shape around a specified center (cx, cy) by a given angle

    How do translation and rotation work? - Translation: Moves the shape by (dx, dy). - Rotation: Rotates the shape around a specified center (cx, cy) by a given angle

  56. [64]

    First, I will circle xx... BEGIN ... END. Hmm, it looks like I have drawn... Then I will... BEGIN ... END

    How do I delete a shape? Use the delete command with the shape’s identifier. For example: delete l1 C Detailed Process of Sketch-CoT Data Synthesis As shown in Figure 6, we present the data synthesis process during the cold start stage. The process begins with the synthesis of...

  57. [65]

    former part,

    There is no legend area annotation, so do not use rectangle to draw the legend area. Instead, use colors to describe the legend. 3. Do not reveal that you can see annotations or metadata. Your output should be a string that simulates a human reasoning dialogue, with no additio...

    1993

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.