REVIEW 3 major objections 6 minor 1 cited by
Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Explicit arrow-direction encoding raises VLM accuracy on flowchart question answering from 80% to 89% and to 100% on next-step queries.
desk verdict A reasonable system paper with a real but tiny measurement whose headline attribution to arrow-direction encoding is not isolated by the experiment; fixable with one ablation, but as written the claim outruns the data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the coordinate-rich graph-structured prompt: each flowchart entity is serialised as (text, category, normalised centre-of-mass) and each object's neighbours are listed as preceding and subsequent steps derived from the recovered arrow endpoints. Arrow direction is made explicit by fine-tuned detection of tiny 'Arrow Start' and 'Arrow End' boxes, which are matched to Arrow boxes by near-edge location and IoU > 0.5, and then used to compute incoming and outgoing links. The paper argues this gives the otherwise text-blind language model a geometry channel—a way to see who points at whom—so that it infers edge orientation from spatial coordinates rather than hallucinating it from an unordered token list.
What would settle it
Run the same 90-question benchmark with the full pipeline but strip only the arrow-derived information—the 'Arrow Start'/'Arrow End' coordinates and the preceding/subsequent links—while keeping OCR text and object boxes. If accuracy stays at 89%, the claim that arrow-direction encoding is what drives the gain is refuted; if it falls back toward 80%, the claim survives.
Extended reading notes
Core claim
The central discovery, stated on the paper's own terms, is that recovering the directional skeleton of a flowchart before reasoning lets a general-purpose vision-language model answer diagram questions it otherwise misses. The proposed pipeline performs OCR, detects nine classes of flowchart objects including small boxes around arrow starts and arrow ends, fuses each text box with the object it overlaps by more than 50%, links arrows to their endpoints by near-edge proximity and a 0.5 IoU criterion, links non-arrow objects to arrows, and then serialises every entity as a category, text, and normalised centre-of-mass coordinate in a graph-structured prompt that lists each object's preceding and subsequent steps. Fed this prompt together with the original image, the model reaches 80/90 (88.9%) overall versus 72/90 (80.0%) for the image-only baseline; next-step questions improve from 25/30 to 30/30, conditional-branch questions from 41/50 to 45/50, and large-diagram accuracy from 66.7% to 80.0%. Preceding-step questions remain difficult, with the baseline at 60% and the pipeline at 50%.
Load-bearing premise
The paper attributes the accuracy gain to arrow-direction encoding, but the full pipeline differs from the baseline by adding OCR text, detected object boxes, coordinate tuples, and graph structure all at once, so the reported experiments do not isolate arrows as the cause of the improvement.
Editorial extensions
If this is right
- Next-step understanding of clean, printed flowcharts at this scale is effectively solved: 30/30 on such queries with the proposed pipeline.
- Diagram QA can be improved modularly—by improving the detector or OCR, not the language model—so better small-object detection of arrowheads should translate directly into higher task accuracy.
- The pipeline transfers the burden of visual parsing out of the VLM, so the same reasoning model can be pointed at new diagram types (e.g., BPMN or UML) by retraining only the detector and OCR fusion, not the language model.
- On large, arrow-dense diagrams the gain is largest (+13.3 pp), suggesting geometry-aware prompts matter most exactly where end-to-end VLMs fail.
Reading between the lines
- Because the comparison adds several variables at once, the most credible reading of the +9 pp is that supplying readable text plus explicit structure together helps; which component carries the weight is untested, and an OCR-without-arrows ablation would settle it.
- On 30 next-step questions, 100% accuracy has a wide confidence interval; the true next-step performance on a larger or messier corpus is likely lower, especially for handwritten diagrams or nodes with multiple incoming edges, which the paper itself flags as a residual error source.
- If arrow recovery is indeed the driver, then detector precision on tiny arrowheads (reported mAP 0.2349 at standard IoU) is the bottleneck, and improving that metric should yield outsized QA gains relative to prompt tweaks.
- The near-zero LLM-judge advantage on conditional-branch questions, despite a human-judged +8 pp gain, suggests part of the measured improvement depends on paraphrase-tolerant evaluation; a strict automatic judge may underestimate genuine gains on open-ended questions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a seven-stage inference pipeline for flowchart question answering: OCR (Azure AI Document Intelligence), fine-tuned DAMO-YOLO object detection with arrow-start/arrow-end classes, text-object association, arrow-start/end linking, node-arrow linking, graph-structured prompt construction, and GPT-4o reasoning. The authors evaluate the pipeline on 90 questions derived from 30 manually annotated flowcharts, covering next-step, conditional-branch, and previous-step question types, by comparing an OCR+detection+graph-prompt condition against a raw-image GPT-4o baseline. They report an overall accuracy gain from 80.0% (72/90) to 88.9% (80/90), perfect 100% on the 30 next-step questions, per-size gains on Large and Medium diagrams, and LLM-as-judge results as a secondary check. The paper explicitly acknowledges the small test set and the dependence on OCR and detector quality.
Significance. If the central claim were established, the paper would offer a simple, modular recipe (explicit coordinates and arrow topology in the prompt) that improves VLM performance on diagram reasoning without VLM fine-tuning, plus a small annotated benchmark for flowchart QA. The paper is transparent about raw counts, reports per-category detection performance in the appendix, and states its limitations clearly. However, the current experimental design does not isolate the arrow-direction-encoding mechanism from the simultaneous addition of OCR text, bounding-box coordinates, and graph structure, and the 'statistically significant' claim is not supported by any reported statistical procedure. A direction-reversal or arrow-removal ablation, paired significance tests, and a better-documented human evaluation protocol are needed before the title-level claim can be considered established.
major comments (3)
- [§4, Tables 2–4] The reported +9 pp gain cannot be attributed to arrow-direction encoding, because the Ocr-Dec condition and the No-Ocr-Dec baseline differ simultaneously in OCR text, detected boxes and coordinates, arrow-start/end linking, and graph-structured prompting. No condition removes only the arrow information; a condition with the same OCR and detection outputs but with arrow edges shuffled, reversed, or omitted is needed. The per-type numbers worsen this concern: Type 3 actually degrades (5/10 vs. 6/10), and the aggregate improvement rests on a handful of Type 1 and Type 2 responses. Please add at least one ablation that isolates arrow direction and discuss the per-type results in that light.
- [§4.2–4.3, abstract] The abstract calls the +9 pp gain 'statistically significant,' but no statistical test, confidence interval, or paired analysis is reported. For independent proportions, 80/90 vs. 72/90 gives p approximately 0.10, and the paired structure (the same 30 diagrams are used in both conditions) requires McNemar's test, for which the discordant counts are not given. The Type 3 cell has only 10 questions and the Type 2 cell has 50; the per-type deltas are +5, +4, and -1. Please report exact paired tests and confidence intervals, or remove the significance claim and describe the improvement as an observed gain on a small sample.
- [§4.2] The primary human-evaluation protocol is described in only one sentence ('correctness was determined by comparing the predicted objectB in the flowchart with the ground-truth object described as A is B'), with no information about the number of annotators, whether they were blinded to condition, the instructions they received, or inter-annotator agreement. Because the LLM-as-judge check uses GPT-4o, the same model family as the system under test, it cannot serve as an independent validation, so the human numbers carry the entire weight of the comparison. Please document the human evaluation protocol in full and report agreement statistics.
minor comments (6)
- [§3.2–3.3] The Text class is listed among the nine annotated classes, but §3.3 says the final implementation relies exclusively on the OCR service for text information; please clarify whether Text detections are used at all and why the class is retained in the annotation scheme.
- [§3.3–3.4] The 50% text-object overlap threshold and the IoU > 0.5 arrow-association threshold are hand-set and not ablated; please add a sensitivity analysis or justify the values empirically, since the reconstructed graph structure depends directly on these thresholds.
- [Table 1 and Table 3] Table 1 states that each diagram is paired with three types of questions, but Table 3 shows per-type denominators of 30, 50, and 10 for Types 1, 2, and 3; please explain how the 90-question total is allocated across question types and diagrams.
- [§4.4] The discussion attributes the discrepancy between human and LLM scores on Type 2 to the LLM judge penalizing paraphrases and extended explanations, but no examples are provided; please include representative outputs to support this claim.
- [§3.7, Appendix B] The exact graph-structured prompt template is not shown; Figure 1 gives an overview, but the precise serialization of (text, x, y, incoming, outgoing) tuples is essential for reproducibility and should be included in an appendix.
- [References and data availability] The annotated benchmark and the code are not released, and some references have incomplete metadata; since the paper introduces a benchmark, releasing the data and annotations would substantially strengthen reproducibility.
Circularity Check
No significant circularity: the central accuracy result is anchored to external human-annotated ground truth, and no prediction reduces to a fitted input or to a self-citation.
full rationale
The paper's central empirical claim is an accuracy comparison on a 90-question benchmark with human-annotated ground truth. The proposed Ocr-Dec pipeline is evaluated against a raw-image baseline; neither condition's outcome is constructed from the other or from the metric. The pipeline's parameters (OCR, DAMO-YOLO, IoU assignment thresholds, prompt serialization) are fixed design choices, not fitted to the 90 test answers, and no quantity that is later 'predicted' appears as a fitted input in an earlier stage. The paper contains no load-bearing self-citations: all cited works are external, and no 'uniqueness theorem' or ansatz is imported from the authors' prior work. The comparison's main weakness is attributional, not circular: Ocr-Dec differs from the baseline along multiple dimensions (OCR text, detected boxes, coordinates, graph-structured prompt), so the +9 pp cannot be uniquely assigned to arrow-direction encoding; and the LLM-as-a-Judge side-check uses GPT-4o to grade GPT-4o outputs, making it a non-independent supplementary signal. These are experimental-validity concerns, not reductions of a prediction to its inputs. Because the primary evaluation is external and human-judged, the derivation chain is not circular.
Assumptions & free parameters
free parameters (3)
- Arrow-Start/End association IoU threshold =
0.5
- Text-object overlap threshold =
50%
- Diagram size cutoffs =
13 and 22 arrows
assumptions (5)
- domain assumption GPT-4o can recover edge orientations from normalized center-of-mass coordinates using geometric priors learned during pre-training.
- domain assumption Azure AI Document Intelligence OCR is accurate enough on printed flowchart text that OCR errors do not dominate the QA result.
- domain assumption A DAMO-YOLO detector fine-tuned on 69 annotated flowcharts generalizes to the 30 test flowcharts.
- domain assumption Human judgment is a reliable ground-truth label without significant annotator disagreement or bias.
- domain assumption Arrow endpoints that satisfy the IoU and edge-proximity heuristics represent the true source and target of each edge.
Cite this review
Pith. "Pith review of Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding." pith.science (2026). https://pith.science/paper/WGZ22FCJ
@misc{pith2026250507864,
author = {Pith},
title = {Pith review of: Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding},
year = {2026},
howpublished = {\url{https://pith.science/paper/WGZ22FCJ}},
note = {Machine review of arXiv:2505.07864}
}
read the original abstract
Flowcharts are indispensable tools in software design and business-process analysis, yet current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology that set these diagrams apart from natural images. We introduce a seven-stage pipeline grouped into three broader processes: (1) arrow-aware detection of nodes and arrow endpoints; (2) optical character recognition (OCR) to extract node text; and (3) construction of a structured prompt that guides the VLMs. Tested on a 90-question benchmark distilled from 30 annotated flowcharts, the method raises overall accuracy from 80 % to 89 % (+9 percentage points) without any task-specific fine-tuning. The gain is most pronounced for next-step queries (25/30 -> 30/30; 100 %, +17 pp); branch-result questions improve more modestly, and before-step questions remain difficult. A parallel evaluation with an LLM-as-a-Judge protocol shows the same trends, reinforcing the advantage of explicit arrow encoding. Limitations include dependence on detector and OCR precision, the small evaluation set, and residual errors at nodes with multiple incoming edges. Future work will enlarge the benchmark with synthetic and handwritten flowcharts and assess the approach on Business Process Model and Notation (BPMN) and Unified Modeling Language (UML).
Figures
Forward citations
Cited by 1 Pith paper
-
Survey of GenAI for Automotive Software Development: From Requirements to Executable Code
A review of roughly 60 papers and 9 industry respondents finds GPT-family models dominate automotive code generation while requirements handling lags due to confidentiality constraints.
Reference graph
Works this paper leans on
-
[1]
D. Hooshyar, R.B. Ahmad, M. Yousefi, F.D. Yusop, and S.-J. Horng. A flowchart-based intelligent tutoring system for improving problem-solving skills of novice programmers. Journal of Computer Assisted Learning, 31(4):345–361, apr 2015. ISSN 1365-2729. doi: 10.1111/jcal.12099. URL http://dx.doi.org/10.1111/jcal.12099
-
[2]
Towards makingflowchartimagesmachineinterpretable
Shreya Shukla, Prajwal Gatti, Yogesh Kumar, Vikash Yadav, and Anand Mishra. Towards makingflowchartimagesmachineinterpretable. 2025. URL http://arxiv.org/pdf/2501. 17441
work page 2025
-
[3]
Measuring massive multitask language understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2020. URLhttp: //arxiv.org/pdf/2009.03300
arXiv 2020
-
[4]
Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and OTHERS
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and OTHERS. Gpt-4o system card. 2024. URL http://arxiv.org/pdf/2410.21276
arXiv 2024
-
[5]
https://openai.com/index/hello-gpt-4o/, 2025
Hello gpt-4o | openai. https://openai.com/index/hello-gpt-4o/, 2025. Accessed: 2025-04-30
work page 2025
-
[6]
Mmmu: Amassivemulti-discipline multimodal understanding and reasoning benchmark for expert agi
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, DongfuJiang,WeimingRen,YuxuanSun,andOTHERS. Mmmu: Amassivemulti-discipline multimodal understanding and reasoning benchmark for expert agi. 2023. URLhttp:// arxiv.org/pdf/2311.16502
arXiv 2023
-
[7]
Mathvista: Evaluatingmathematical reasoning of foundation models in visual contexts
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng,Kai-WeiChang,MichelGalley,andJianfengGao. Mathvista: Evaluatingmathematical reasoning of foundation models in visual contexts. 2023. URLhttp://arxiv.org/pdf/ 2310.02255
arXiv 2023
-
[8]
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. 2020. URLhttp://arxiv.org/pdf/2007.00398
arXiv 2020
Show all 28 references
-
[9]
FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding
HuitongPan,QiZhang,CorneliaCaragea,EduardDragut,andLonginJanLatecki. FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding. IOS Press, oct 2024. ISBN 9781643685489. doi: 10.3233/faia240473. URL http://dx.doi.org/10.3233/ FAIA240473
2024 doi
-
[10]
Youonlylookonce: Unified, real-time object detection
JosephRedmon,SantoshDivvala,RossGirshick,andAliFarhadi. Youonlylookonce: Unified, real-time object detection. 2015. URLhttp://arxiv.org/pdf/1506.02640. 8
2015 arXiv
-
[11]
Berg, Wan-Yen Lo, and OTHERS
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, and OTHERS. Segment anything. 2023. URLhttp://arxiv.org/pdf/2304.02643
2023 arXiv
-
[12]
GenFlowchart: Parsing and Understanding Flowchart Using Generative AI, page 99–111
AbdulArbaz,HengFan,JunhuaDing,MeikangQiu,andYunheFeng. GenFlowchart: Parsing and Understanding Flowchart Using Generative AI, page 99–111. Springer Nature Singapore,
-
[13]
Arrow r-cnn for handwritten diagram recognition
Bernhard Schäfer, Margret Keuper, and Heiner Stuckenschmidt. Arrow r-cnn for handwritten diagram recognition. International Journal on Document Analysis and Recognition (IJDAR), 24(1–2):3–17, feb 2021. ISSN 1433-2825. doi: 10.1007/s10032-020-00361-1. URLhttp: //dx.doi.org/10.10...
2021 doi
-
[14]
Are we on the right way for evaluating large vision-language models? 2024
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and OTHERS. Are we on the right way for evaluating large vision-language models? 2024. URLhttp://arxiv.org/pdf/2403.20330
2024 arXiv
-
[15]
A diagram is worth a dozen images
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. 2016. URLhttp://arxiv.org/pdf/1603. 07396
2016
-
[16]
Chartinsights: Evaluating multimodal large language models for low-level chart question answering
Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. Chartinsights: Evaluating multimodal large language models for low-level chart question answering. 2024. URL http://arxiv.org/pdf/2405.07001
2024 arXiv
-
[17]
Charxiv: Charting gaps in realistic chart understanding in multimodal llms
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, and OTHERS. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. 2024. URLhttp://arxiv.org/pdf/2406.18521
2024 arXiv
-
[18]
Chartx & chartvlm: A versatile benchmark andfoundationmodelforcomplicatedchartreasoning
Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Peng Ye, Min Dou, Botian Shi, and OTHERS. Chartx & chartvlm: A versatile benchmark andfoundationmodelforcomplicatedchartreasoning. 2024. URL http://arxiv.org/pdf/ 2402.12185
2024 arXiv
-
[19]
Shikra: Unleashing multimodal llm’s referential dialogue magic
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. 2023. URLhttp://arxiv.org/ pdf/2306.15195
2023 arXiv
-
[20]
Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning
LiangZhao,EnYu,ZhengGe,JinrongYang,HaoranWei,HongyuZhou,JianjianSun,Yuang Peng, Runpei Dong, Chunrui Han, and OTHERS. Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. 2023. URLhttp://arxiv.org/pdf/2307.09474
2023 arXiv
-
[21]
Region- blip: A unified multi-modal pre-training framework for holistic and regional comprehension
QiangZhou,ChaohuiYu,ShaofengZhang,SitongWu,ZhibingWang,andFanWang. Region- blip: A unified multi-modal pre-training framework for holistic and regional comprehension
-
[22]
Grcnn: Graph recognition convolutional neural network for synthesizingprogramsfromflowcharts
Lin Cheng and Zijiang Yang. Grcnn: Graph recognition convolutional neural network for synthesizingprogramsfromflowcharts. 2020. URL http://arxiv.org/pdf/2011.05980
2020 arXiv
-
[23]
Damo-yolo : A report on real-time object detection design
XianzheXu,YiqiJiang,WeihuaChen,YilunHuang,YuanZhang,andXiuyuSun. Damo-yolo : A report on real-time object detection design. 2022. URLhttp://arxiv.org/pdf/2211. 15444. A Additional Evaluation Results We provide here additional results and analysis that complement the main paper...
2022
-
[26]
A question (Question)
-
[27]
A reference answer (Reference Answer)
-
[28]
- Determine whether they essentially match in meaning or reasoning, or if the model output is otherwise correct based on its logic and evidence
A model output (Model Output) Please evaluate the model output by following these steps: ### Step 1: Analyze the Answers - First, compare the reference answer and the model output. - Determine whether they essentially match in meaning or reasoning, or if the model output is ot...
-
[2023]
URL http://arxiv.org/pdf/2308.02299
-
[2024]
doi: 10.1007/978-981-97-5492-2_8
ISBN 9789819754922. doi: 10.1007/978-981-97-5492-2_8. URL http://dx.doi. org/10.1007/978-981-97-5492-2_8
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.