{"id":"c53dea78-9ab7-47a0-93d8-be17d0f2c4e1","arxiv_id":"2505.07864","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A seven-stage OCR plus arrow-detection prompt pipeline lifts GPT-4o flowchart QA accuracy from 80% to 89% on 90 self-annotated questions, without task-specific fine-tuning.","lead":"Flowchart questions trip up vision-language models because arrows and topology are easy to miss. This paper adds OCR, arrow detection, and a structured text prompt before asking GPT-4o, lifting accuracy from 80% to 89% on a 90-question private benchmark; the arrow-specific gain is not yet isolated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gain is not attributable to arrow-direction encoding because Ocr-Dec differs from the baseline by OCR text, box coordinates, and graph structure; no ablation isolates arrow direction, and the claimed significance is unsupported.","rationale":"The reader's weakest assumption correctly identifies the central gap. The paper's own framing in the title, abstract, and conclusion makes the causal attribution to arrow-direction encoding load-bearing, and every reported comparison changes multiple pipeline components at once. Existing work cited in Section 2.2 already shows that simply adding OCR text and boxes to prompts can yield gains, so the +9 pp could be explained without arrow direction at all. The additional 'statistically significant' claim in the abstract is unsupported by any test in the paper; a crude independent-proportion calculation on 80/90 vs 72/90 gives p approximately 0.10, and Type 3 with n=10 actually moves against the method. The concrete experiment that would settle the attribution is a matched ablation that removes only edge orientation. Because the system-level result may survive such a test, this is a conditional-acceptance issue rather than a rejection; the reader's CONDITIONAL verdict is therefore the right call and needs no change.","tokens_in":8825,"tokens_out":4405,"duration_ms":45936,"concrete_test":"Run a third condition, Ocr-Dec-unoriented, that uses identical OCR text, detected boxes, and coordinate serialization but replaces each directed edge (A→B) in the structured prompt with an unordered pair {A,B}, or randomly permutes the start/end labels, leaving all other pipeline stages fixed. Evaluate on the same 90 questions with the same GPT-4o settings and the same human and LLM scoring protocols, then perform a paired McNemar test between Ocr-Dec and Ocr-Dec-unoriented. If accuracy stays near 88.9%, arrow direction contributes nothing beyond the other components; if it drops toward the 80% baseline, the title claim is supported. Also report the discordant pair counts and a confidence interval for the original 9 pp difference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's title and abstract attribute the +9 pp gain to explicit arrow-direction encoding, but Tables 2-4 compare a full pipeline (OCR text + detected boxes + coordinate tuples + graph-structured prompt) against a raw-image baseline. These conditions differ along at least four dimensions simultaneously, and no condition removes only arrow direction. The improvement could therefore come entirely from supplying OCR text or object boxes and coordinates, both of which prior work in Section 2.2 already found to give modest gains. The Type 1 and Type 2 gains are only 5 and 4 questions respectively, while Type 3 actually degrades (5/10 vs 6/10), so the aggregate +9 pp rests on a small number of responses. The abstract calls the gain statistically significant, but no test or confidence interval is reported; a rough independent-proportion z-test on 80/90 vs 72/90 gives p approximately 0.10, and the paired discordant counts needed for McNemar are absent. With n=90 and a 10-question Type 3 cell, the empirical basis for the headline claim is fragile. Without an ablation that removes only arrow-direction information, the central explanation remains an attribution, not an established mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a seven-stage inference pipeline for flowchart question answering: OCR (Azure AI Document Intelligence), fine-tuned DAMO-YOLO object detection with arrow-start/arrow-end classes, text-object association, arrow-start/end linking, node-arrow linking, graph-structured prompt construction, and GPT-4o reasoning. The authors evaluate the pipeline on 90 questions derived from 30 manually annotated flowcharts, covering next-step, conditional-branch, and previous-step question types, by comparing an OCR+detection+graph-prompt condition against a raw-image GPT-4o baseline. They report an overall accuracy gain from 80.0% (72/90) to 88.9% (80/90), perfect 100% on the 30 next-step questions, per-size gains on Large and Medium diagrams, and LLM-as-judge results as a secondary check. The paper explicitly acknowledges the small test set and the dependence on OCR and detector quality.","tokens_in":9064,"tokens_out":5899,"duration_ms":60709,"significance":"If the central claim were established, the paper would offer a simple, modular recipe (explicit coordinates and arrow topology in the prompt) that improves VLM performance on diagram reasoning without VLM fine-tuning, plus a small annotated benchmark for flowchart QA. The paper is transparent about raw counts, reports per-category detection performance in the appendix, and states its limitations clearly. However, the current experimental design does not isolate the arrow-direction-encoding mechanism from the simultaneous addition of OCR text, bounding-box coordinates, and graph structure, and the 'statistically significant' claim is not supported by any reported statistical procedure. A direction-reversal or arrow-removal ablation, paired significance tests, and a better-documented human evaluation protocol are needed before the title-level claim can be considered established.","major_comments":[{"comment":"The reported +9 pp gain cannot be attributed to arrow-direction encoding, because the Ocr-Dec condition and the No-Ocr-Dec baseline differ simultaneously in OCR text, detected boxes and coordinates, arrow-start/end linking, and graph-structured prompting. No condition removes only the arrow information; a condition with the same OCR and detection outputs but with arrow edges shuffled, reversed, or omitted is needed. The per-type numbers worsen this concern: Type 3 actually degrades (5/10 vs. 6/10), and the aggregate improvement rests on a handful of Type 1 and Type 2 responses. Please add at least one ablation that isolates arrow direction and discuss the per-type results in that light.","section":"§4, Tables 2–4"},{"comment":"The abstract calls the +9 pp gain 'statistically significant,' but no statistical test, confidence interval, or paired analysis is reported. For independent proportions, 80/90 vs. 72/90 gives p approximately 0.10, and the paired structure (the same 30 diagrams are used in both conditions) requires McNemar's test, for which the discordant counts are not given. The Type 3 cell has only 10 questions and the Type 2 cell has 50; the per-type deltas are +5, +4, and -1. Please report exact paired tests and confidence intervals, or remove the significance claim and describe the improvement as an observed gain on a small sample.","section":"§4.2–4.3, abstract"},{"comment":"The primary human-evaluation protocol is described in only one sentence ('correctness was determined by comparing the predicted objectB in the flowchart with the ground-truth object described as A is B'), with no information about the number of annotators, whether they were blinded to condition, the instructions they received, or inter-annotator agreement. Because the LLM-as-judge check uses GPT-4o, the same model family as the system under test, it cannot serve as an independent validation, so the human numbers carry the entire weight of the comparison. Please document the human evaluation protocol in full and report agreement statistics.","section":"§4.2"}],"minor_comments":[{"comment":"The Text class is listed among the nine annotated classes, but §3.3 says the final implementation relies exclusively on the OCR service for text information; please clarify whether Text detections are used at all and why the class is retained in the annotation scheme.","section":"§3.2–3.3"},{"comment":"The 50% text-object overlap threshold and the IoU > 0.5 arrow-association threshold are hand-set and not ablated; please add a sensitivity analysis or justify the values empirically, since the reconstructed graph structure depends directly on these thresholds.","section":"§3.3–3.4"},{"comment":"Table 1 states that each diagram is paired with three types of questions, but Table 3 shows per-type denominators of 30, 50, and 10 for Types 1, 2, and 3; please explain how the 90-question total is allocated across question types and diagrams.","section":"Table 1 and Table 3"},{"comment":"The discussion attributes the discrepancy between human and LLM scores on Type 2 to the LLM judge penalizing paraphrases and extended explanations, but no examples are provided; please include representative outputs to support this claim.","section":"§4.4"},{"comment":"The exact graph-structured prompt template is not shown; Figure 1 gives an overview, but the precise serialization of (text, x, y, incoming, outgoing) tuples is essential for reproducibility and should be included in an appendix.","section":"§3.7, Appendix B"},{"comment":"The annotated benchmark and the code are not released, and some references have incomplete metadata; since the paper introduces a benchmark, releasing the data and annotations would substantially strengthen reproducibility.","section":"References and data availability"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim needs an ablation that isolates arrow direction plus proper paired statistics; without those, the paper currently overstates what is demonstrated. The contribution is also modest in scope, so even after revision the editors should weigh whether the depth of analysis is sufficient for a full journal article."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing about this paper: the +9 pp gain it reports is probably real as a system-level result, but the title's claim that arrow-direction encoding causes the gain is not supported by the experiments. The stress-test note lands. The treatment differs from the baseline by OCR text, detected boxes, coordinate tuples, and graph structure all at once. No condition removes only the arrow information, so the improvement could come from any of those components. Prior work already shows that adding OCR and boxes helps by itself, so the attribution is genuinely unestablished. On top of that, the aggregate +9 pp rests on eight questions, Type 3 actually gets worse (5/10 vs 6/10), and the abstract calls the gain statistically significant despite no test or confidence interval being reported. A rough independent-proportion z-test gives p around 0.10. With n=90 and a 10-question cell, that is fragile.\n\nCredit where it is due. The pipeline itself is clean and modular: fine-tuned DAMO-YOLO with separate Arrow Start/End classes, Azure OCR, IoU-based arrow association, and a coordinate-rich graph prompt fed to GPT-4o. That integration is new in the flowchart domain even if each ingredient exists. The authors are honest about the small set and detector dependence, and they include detection AP numbers in the appendix, which is more than most such papers do. The internal arithmetic is consistent across Tables 2-4, and the human evaluation is described clearly enough to follow.\n\nThe soft spots beyond the missing ablation: the benchmark is private with no code or data release, so the numbers are not independently checkable. The LLM-as-judge protocol uses GPT-4o to score GPT-4o output, though the authors sensibly make human judgment primary. The Type 2 discrepancy between human and LLM scoring (90 vs 84) is not resolved, and there is no inter-annotator reliability on the human side. None of these are fatal, but all are real.\n\nThe citation pattern looks fine. They cite the relevant flowchart and coordinate-prompting work and do not oversell related work.\n\nWho is this for? People working on diagram understanding, document VQA, or prompt-engineering pipelines for structured visual input. It is a niche system paper, not a method paper.\n\nMy recommendation: do not desk-reject, but send to peer review with a narrow mandate. The key revision is one ablation that keeps OCR and detection but removes arrow direction, plus confidence intervals or a significance test, and ideally data or code. If the authors cannot add the ablation, the paper should be reframed as a system report and drop the causal claim in the title.","headline":"A reasonable system paper with a real but tiny measurement whose headline attribution to arrow-direction encoding is not isolated by the experiment; fixable with one ablation, but as written the claim outruns the data.","tokens_in":9604,"tokens_out":2182,"would_cite":false,"duration_ms":26236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explicit arrow-direction encoding raises VLM accuracy on flowchart question answering from 80% to 89% and to 100% on next-step queries.","keywords":["flowchart understanding","vision-language models","arrow direction encoding","graph-structured prompting","object detection","OCR","diagram question answering","graph topology"],"falsifier":"Run the same 90-question benchmark with the full pipeline but strip only the arrow-derived information—the 'Arrow Start'/'Arrow End' coordinates and the preceding/subsequent links—while keeping OCR text and object boxes. If accuracy stays at 89%, the claim that arrow-direction encoding is what drives the gain is refuted; if it falls back toward 80%, the claim survives.","tokens_in":8617,"feed_emoji":"➡️","tokens_out":7446,"duration_ms":68525,"temperature":0.7,"pith_summary":"This paper claims that a vision-language model can answer questions about flowcharts far more reliably when the diagram is first parsed by a physical pipeline—optical character recognition plus a fine-tuned object detector that explicitly localises arrow starts and ends—and the parsed information is handed to the model as a structured, coordinate-rich text prompt rather than as pixels alone. On a 90-question benchmark built from 30 annotated flowcharts, overall accuracy rises from 80% to 89% with no task-specific fine-tuning of the language model, and 'what is the next step?' questions are answered correctly 100% of the time. The authors interpret the gains as evidence that explicit arrow-direction and graph-topology encoding closes a reasoning gap that end-to-end vision-language models leave open. The claim matters because it suggests a modular route to making process diagrams, engineering schematics, and similar structured images queryable by today's language models without retraining them.","feed_headline":"Arrow directions lift flowchart QA accuracy from 80% to 89%","feed_subtitle":"A detector-plus-OCR pipeline feeds the VLM coordinate-rich prompts, hitting 100% on next-step questions.","key_machinery":"The load-bearing mechanism is the coordinate-rich graph-structured prompt: each flowchart entity is serialised as (text, category, normalised centre-of-mass) and each object's neighbours are listed as preceding and subsequent steps derived from the recovered arrow endpoints. Arrow direction is made explicit by fine-tuned detection of tiny 'Arrow Start' and 'Arrow End' boxes, which are matched to Arrow boxes by near-edge location and IoU > 0.5, and then used to compute incoming and outgoing links. The paper argues this gives the otherwise text-blind language model a geometry channel—a way to see who points at whom—so that it infers edge orientation from spatial coordinates rather than hallucinating it from an unordered token list.","core_discovery":"The central discovery, stated on the paper's own terms, is that recovering the directional skeleton of a flowchart before reasoning lets a general-purpose vision-language model answer diagram questions it otherwise misses. The proposed pipeline performs OCR, detects nine classes of flowchart objects including small boxes around arrow starts and arrow ends, fuses each text box with the object it overlaps by more than 50%, links arrows to their endpoints by near-edge proximity and a 0.5 IoU criterion, links non-arrow objects to arrows, and then serialises every entity as a category, text, and normalised centre-of-mass coordinate in a graph-structured prompt that lists each object's preceding and subsequent steps. Fed this prompt together with the original image, the model reaches 80/90 (88.9%) overall versus 72/90 (80.0%) for the image-only baseline; next-step questions improve from 25/30 to 30/30, conditional-branch questions from 41/50 to 45/50, and large-diagram accuracy from 66.7% to 80.0%. Preceding-step questions remain difficult, with the baseline at 60% and the pipeline at 50%.","pith_inferences":["Because the comparison adds several variables at once, the most credible reading of the +9 pp is that supplying readable text plus explicit structure together helps; which component carries the weight is untested, and an OCR-without-arrows ablation would settle it.","On 30 next-step questions, 100% accuracy has a wide confidence interval; the true next-step performance on a larger or messier corpus is likely lower, especially for handwritten diagrams or nodes with multiple incoming edges, which the paper itself flags as a residual error source.","If arrow recovery is indeed the driver, then detector precision on tiny arrowheads (reported mAP 0.2349 at standard IoU) is the bottleneck, and improving that metric should yield outsized QA gains relative to prompt tweaks.","The near-zero LLM-judge advantage on conditional-branch questions, despite a human-judged +8 pp gain, suggests part of the measured improvement depends on paraphrase-tolerant evaluation; a strict automatic judge may underestimate genuine gains on open-ended questions."],"forward_implications":["Next-step understanding of clean, printed flowcharts at this scale is effectively solved: 30/30 on such queries with the proposed pipeline.","Diagram QA can be improved modularly—by improving the detector or OCR, not the language model—so better small-object detection of arrowheads should translate directly into higher task accuracy.","The pipeline transfers the burden of visual parsing out of the VLM, so the same reasoning model can be pointed at new diagram types (e.g., BPMN or UML) by retraining only the detector and OCR fusion, not the language model.","On large, arrow-dense diagrams the gain is largest (+13.3 pp), suggesting geometry-aware prompts matter most exactly where end-to-end VLMs fail."],"supporting_citations":[{"why":"Documents that leading vision-language models score only 0.22-0.30 link-level F1 on flowchart-structure recovery, defining the gap the method targets.","marker":"[9]"},{"why":"The real-time detector the authors fine-tune to localise flowchart nodes and arrow endpoints.","marker":"[23]"},{"why":"Prior evidence that arrow-aware detection halves localisation error on diagram inputs, motivating the small arrow-start/arrow-end boxes.","marker":"[13]"},{"why":"A representative detector-plus-language-model pipeline that still misorders branches, which the new graph-structured prompt is meant to overcome.","marker":"[12]"},{"why":"Shows explicit coordinate tokens help a multimodal model refer to specific image regions, grounding the coordinate-rich serialisation.","marker":"[19]"},{"why":"Provides the end-to-end structure-prediction alternative, contrasted with the modular detector-VLM design.","marker":"[2]"}],"fun_headline_variants":["Arrow encoding boosts flowchart VLM accuracy to 89%","Direction-aware pipeline lifts flowchart QA to 89%","Arrow-guided prompts hit 100% on next-step queries","Explicit arrow directions improve VLM flowchart reasoning","Flowchart VLM gains 9 points from arrow-aware prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper attributes the accuracy gain to arrow-direction encoding, but the full pipeline differs from the baseline by adding OCR text, detected object boxes, coordinate tuples, and graph structure all at once, so the reported experiments do not isolate arrows as the cause of the improvement.","fun_headline_variants_meta":{"raw":{"variants":["Arrow encoding boosts flowchart VLM accuracy to 89%","Direction-aware pipeline lifts flowchart QA to 89%","Arrow-guided prompts hit 100% on next-step queries","Explicit arrow directions improve VLM flowchart reasoning","Flowchart VLM gains 9 points from arrow-aware prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1280,"prompt_tokens":1015,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":631,"tokens_out":265,"duration_ms":3170,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:56:15.034642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 90-question benchmark with the full pipeline but strip only the arrow-derived information—the 'Arrow Start'/'Arrow End' coordinates and the preceding/subsequent links—while keeping OCR text and object boxes. If accuracy stays at 89%, the claim that arrow-direction encoding is what drives the gain is refuted; if it falls back toward 80%, the claim survives.","supporting_citations":[{"cited_title":"FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding","cited_arxiv_id":null,"evidence_quote":"Documents that leading vision-language models score only 0.22-0.30 link-level F1 on flowchart-structure recovery, defining the gap the method targets."},{"cited_title":"Damo-yolo : A report on real-time object detection design","cited_arxiv_id":null,"evidence_quote":"The real-time detector the authors fine-tune to localise flowchart nodes and arrow endpoints."},{"cited_title":"Arrow r-cnn for handwritten diagram recognition","cited_arxiv_id":null,"evidence_quote":"Prior evidence that arrow-aware detection halves localisation error on diagram inputs, motivating the small arrow-start/arrow-end boxes."},{"cited_title":"GenFlowchart: Parsing and Understanding Flowchart Using Generative AI, page 99–111","cited_arxiv_id":null,"evidence_quote":"A representative detector-plus-language-model pipeline that still misorders branches, which the new graph-structured prompt is meant to overcome."},{"cited_title":"Towards makingflowchartimagesmachineinterpretable","cited_arxiv_id":null,"evidence_quote":"Provides the end-to-end structure-prediction alternative, contrasted with the modular detector-VLM design."}],"review_version":1}