Pith. sign in

REVIEW 3 major objections 6 minor 28 references

Frontier vision-language models fail on long multi-page documents in three controlled ways existing benchmarks hide.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:01 UTC pith:DIG4MUSC

load-bearing objection Solid controlled diagnostic for long-context visual docs: three real failure modes, released code/data, and confounds the authors already flag. the 3 major comments →

arxiv 2607.10400 v1 pith:DIG4MUSC submitted 2026-07-11 cs.CV cs.AI

SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Abhigya Verma , Khyati Mahajan , Amit Kumar Saha , Shruthan Radhakrishna , Sagar Davasam , Vikas Yadav , Sai Rajeswar Mudumba This is my paper
classification cs.CV cs.AI
keywords vision-language modelslong-context document understandingsynthetic benchmarkchart readingcross-modal reasoningpositional biasvisual document VQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Real document benchmarks mix length, layout, charts, and question difficulty so tightly that a wrong answer rarely tells you which factor broke. SynthDocBench is a fully synthetic long-context suite that varies those factors independently across 200 multi-page reports and 1,788 questions spanning chart reading, cross-modal grounding, and multi-hop reasoning. When seven frontier models are evaluated under a vision-only protocol, three failures appear together: accuracy falls sharply as evidence complexity and reasoning depth rise from L1 to L5; the middle third of a document is hardest for most models and most models lose accuracy from early to late evidence; and precise chart-value reading collapses once charts sit inside long documents rather than as isolated images. The paper argues these patterns show models may be overfitting to the artifacts of existing short or uncontrolled benchmarks rather than achieving genuine long-context visual document understanding.

Core claim

By holding document factors independent in a synthetic long-context suite, the authors show that frontier VLMs exhibit three concurrent, previously unobservable failure modes: sharp degradation with evidence complexity and reasoning depth, systematic positional sensitivity that makes the middle of a document hardest and produces a negative early-to-late trend for most models, and collapse of precise chart-reading accuracy once charts are embedded in multi-page contexts.

What carries the argument

SynthDocBench: an end-to-end LLM generation pipeline that produces dual-layer documents (rendered D3.js charts plus hidden structured metadata for deterministic ground truth) across six layout archetypes with a 40% random override, yielding three controlled question families (chart, cross-modal, complex multi-hop) at difficulty levels L1–L5.

Load-bearing premise

That the measured gaps mainly reflect true long-context visual reasoning limits rather than models’ familiarity with the HTML/D3.js rendering style or quirks of the primary automatic judge.

What would settle it

Re-render the same documents and questions with a non-web chart backend (e.g., vector or scientific plotting) and re-score with multiple independent judges; if the three failure modes shrink or reverse while real-document rankings stay stable, the central diagnostic claim weakens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Progress claims based only on DocVQA, ChartQA, or uncontrolled multi-page suites will systematically overstate long-context visual robustness.
  • Model development must separately target middle-of-document retrieval, pixel-level chart decoding under long context, and multi-hop cross-modal alignment.
  • Future long-document benchmarks can isolate single axes of difficulty instead of confounding them, making failure attribution routine.
  • Vision-only evaluation with deterministic synthetic ground truth becomes a practical alternative to costly real-document annotation for diagnosis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training curricula that never place dense charts deep inside long page sequences may leave a permanent blind spot that isolated chart benchmarks cannot detect.
  • The middle-third hardness pattern is consistent with known long-context retrieval biases and may be mitigated by explicit positional encoding or hierarchical document memory.
  • If rendering-style familiarity proves real, community benchmarks will need multi-backend rendering to prevent silent distribution overfitting.
  • The OCR-versus-vision asymmetry implies hybrid pipelines remain necessary until pure VLMs close the precise chart-value gap inside long documents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SynthDocBench, a fully synthetic, factorially controlled benchmark for long-context visual document understanding. Documents are generated end-to-end via an LLM pipeline over six layout archetypes (with a 40% random layout override), dual-layer D3.js charts plus hidden structured metadata for deterministic ground truth, and three QA families (chart-reading, cross-modal, complex multi-hop) at difficulty levels L1–L5. The released corpus comprises 200 multi-page reports (avg. 51.1 pages, ~20.6k words, 16.7 charts) and 1,788 questions. Evaluating eight frontier VLMs under a vision-only protocol with GPT-5 as primary judge (cross-validated against Gemini), the authors report three concurrent failure modes: degradation with reasoning depth (L1→L5), positional sensitivity (middle third often hardest; several models show Early→Late declines), and collapse of precise chart reading once charts sit in long multi-page contexts—supported by OCR baselines, bootstrap CIs, and rendering/prompting ablations.

Significance. If the diagnostic claims hold, the work fills a clear gap: existing chart and long-document VQA benchmarks either isolate charts or confound length, layout, modality, and question difficulty, so failures cannot be attributed. The dual-layer generation design, combinatorial axes, deterministic answers, public code/dataset, cross-judge checks (GPT-5 vs Gemini r≥0.94, ΔACC≤3.5), OCR vs vision split, and systematic ablations (pages-per-strip, DPI, prompting) are concrete methodological strengths that make the failure modes falsifiable and reusable. The positional and chart-in-context results are especially useful for the community, because they surface deficits that saturated single-page DocVQA/ChartQA scores and heterogeneous real-document suites do not isolate. The contribution is primarily empirical and diagnostic rather than algorithmic, but that is appropriate for a benchmark paper and is executed at a high standard.

major comments (3)
  1. [Abstract; §1; §5 Table 3–4; Conclusion] Abstract vs body mismatch on the first failure mode and on reported statistics. The abstract states three modes including “sharp degradation with document length” and “middle third … hardest for five of six models … negative Early-to-Late trend (steepest decline: 8.3 percentage points),” and “seven frontier VLMs.” Contribution 3 and §5 instead emphasize degradation with evidence complexity/reasoning depth (L1→L5; Table 3), Table 2 evaluates eight models, and Table 4 reports middle-third hardness for 5 of 8 models with Early→Late Δ as large as −11.7 pp (Claude) and −16.0 pp (Qwen3.5-VL-122B). The Conclusion again uses different counts (“four of six” / “three of six”). Please align abstract, intro, and conclusion with the tables, and either (i) add a direct stratification of ACC by document page length (pages range 24–91 in Table 1) under controlled other factors, or (ii) rephrase the firs
  2. [§3.1–3.3; §5; Table 1 vs Tables 2–4] The claim that length is varied as an independent diagnostic axis is only weakly realized in the reported results. §3 and the contributions assert combinatorial control of document length, yet the main empirical sections do not report ACC vs page count (or word count) while holding modality and question type fixed; the closest analyses are positional thirds within documents (Table 4) and pages-per-strip presentation ablations (Table 5), which measure context packing, not document length. Without a length-stratified result, the abstract’s “degradation with document length” and the “independently variable length” framing overclaim relative to the evidence. A short length-bucket table (or explicit statement that length is controlled for generation diversity but not the primary reported axis) is needed for the central diagnostic narrative to be load-bearing as written.
  3. [§5 (Gemini dominance paragraph); Appendix C Tables 8–9] Rendering-familiarity and judge absolute-score sensitivity remain residual threats to interpreting absolute ACC levels, especially Gemini’s large lead. §5 already flags HTML/D3.js familiarity as a possible confound for the Gemini–Qwen gap and shows Claude-as-judge is systematically more lenient (+11–16 ACC overall; Appendix C). Relative rankings and the qualitative existence of positional / L1→L5 / chart-in-context modes are reasonably robust (Gemini-as-judge recovers rankings; OCR isolates visual chart bottleneck). Still, for the stronger claim that models “may be overfitting to benchmark artifacts,” the paper should either (a) report at least one alternative rendering backend or non-web chart style on a subset, or (b) clearly scope the claim to relative patterns under this rendering distribution rather than absolute competence. This is fixable within the manuscript’s scope but currentl
minor comments (6)
  1. [Figure 1] Figure 1 caption and landscape discussion refer to “Top/Bottom” layout that is hard to parse in the text-only rendering; ensure the published figure clearly separates the scatter of length vs tokens from the tabular comparison.
  2. [Table 2] Table 2 header says “Prior Benchmarks” with DocVQA and MMLongBench-Doc, but several models lack published MMLongBench-Doc numbers (—); a footnote on score provenance and access date (already partly in footnote 1) would help.
  3. [§5; Figure 6; Appendix M] Hard-failure analysis text oscillates between “all six models” and “all eight models” (Figure 6 caption vs §5 and Appendix M Table 21). Unify the model set used for the hard-failure definition.
  4. [§4 Eq. (1); Appendix L.1; Appendix N.4] Notation: difficulty is ℓ in the glossary (Table 15) but L1–L5 in the main text; pick one convention. Also clarify whether τ=6 is inherited solely from MMLongBench-Doc or re-validated on this corpus (Appendix N.4 helps but is buried).
  5. [Ethics Statement; Reproducibility Statement] Ethics/reproducibility statements are thorough; consider adding approximate API cost or token volume alongside the stated 150 GPU-hours / 200 API-hours for completeness.
  6. [Throughout] Minor copy-edits: “adual-layer” spacing in §3.1; “SYNTHDOCBENCH” capitalization is inconsistent with “SynthDocBench” in the abstract and URLs; “five of six” leftovers after model-set expansion.

Circularity Check

0 steps flagged

No significant circularity: SynthDocBench is an empirical diagnostic benchmark whose ground truth and failure-mode claims do not reduce to fitted inputs or self-referential definitions.

full rationale

The paper’s load-bearing chain is construction → vision-only evaluation → stratified measurement, not a first-principles derivation that could collapse into its inputs. Documents and questions are generated with a dual-layer design in which chart/table semantics live in structured metadata M used only for deterministic reference answers, while candidate models receive only rendered page images and never M (Sections 3.1–3.2, 4). That is standard synthetic-benchmark practice (explicitly analogous to CLEVR/bAbI), not self-definitional circularity: the measured ACC and judge scores are not forced by the generation parameters. The three claimed failure modes (L1→L5 degradation, positional Early/Middle/Late sensitivity, chart-reading collapse in long multi-page context) are empirical patterns over eight VLMs on a held-out test set; no parameter is fit on a subset and then “predicted” on a closely related quantity, and no uniqueness theorem or ansatz is imported via author-overlapping self-citation to forbid alternatives. External citations (DocVQA, ChartQA, MMLongBench-Doc, lost-in-the-middle, etc.) supply background and comparison, not the existence of the reported modes. Minor abstract/body numerical phrasing differences and the paper’s own Gemini rendering-familiarity caveat are correctness or external-validity concerns, not circular reductions. The derivation is therefore self-contained against its own inputs; score 0 is the honest finding.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

As an empirical benchmark paper the load-bearing commitments are design choices and evaluation conventions rather than physical axioms. The free parameters are the controllable generation knobs and scoring thresholds; the axioms are standard assumptions of synthetic diagnostic design and LLM-as-judge evaluation; the invented entities are the benchmark constructs themselves.

free parameters (5)
  • layout random-override probability = 0.4
    Fixed at 40% to break topic–layout correlations; chosen by design, not derived.
  • ACC correctness threshold τ = 6
    Judge score ≥6 counted as correct, aligned with MMLongBench-Doc but still a free scoring choice that affects absolute ACC.
  • pages-per-strip (concat-num) default = 5
    Default 5-page vertical strips chosen to satisfy API limits; ablations show large sensitivity (Gemini ACC 0.369→0.792 from 1 to 10 pages).
  • rasterization DPI default = 144
    144 DPI selected as optimal under 4 MB JPEG cap; higher DPI degrades under compression.
  • difficulty taxonomy L1–L5 and evidence-unit counts
    Hand-designed difficulty ladder and 2–4 evidence-unit multi-hop definition control the complexity axis.
axioms (4)
  • domain assumption Synthetic factorial control trades ecological validity for causal attribution of failures (CLEVR/bAbI-style principle).
    Stated in §1 and used to justify the entire design; validity of conclusions about real-world robustness rests on this trade-off being informative.
  • domain assumption LLM-as-judge (GPT-5) scores with the disclosed 0–10 rubric are sufficiently reliable proxies for answer correctness.
    Primary metric; partially validated by Gemini re-scoring (r≥0.94) but absolute scores remain judge-dependent.
  • domain assumption Vision-only protocol (page images, no HTML/metadata) is the appropriate evaluation setting for long-context visual document understanding.
    Defines the task; OCR baseline is used only as a diagnostic contrast.
  • standard math Standard mathematical and statistical tools (bootstrap CIs, Spearman/Pearson rank correlation, indicator ACC).
    Used for ranking reliability and external-validity checks.
invented entities (3)
  • SynthDocBench (200 synthetic multi-page reports + 1,788 questions across three subsets) no independent evidence
    purpose: Provide a controlled test distribution that independently varies length, layout, modality, and question type.
    Core contribution; no independent existence outside the paper’s generation pipeline.
  • Dual-layer visualization (D3.js rendering + hidden structured metadata Vk) no independent evidence
    purpose: Make ground-truth answers deterministic without post-hoc chart parsing or human annotation.
    Enables exact numeric recomputation and QA generation; internal to the pipeline.
  • Six layout archetypes (Magazine, Dashboard, Academic, Editorial, Infographic, Brutalist) no independent evidence
    purpose: Control page grammar and chart placement strategy while preserving diversity.
    Design inventory introduced by the authors; 40% random override further randomizes assignment.

pith-pipeline@v1.1.0-grok45 · 30312 in / 3304 out tokens · 34287 ms · 2026-07-14T12:01:07.865813+00:00 · methodology

0 comments
read the original abstract

Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.

Figures

Figures reproduced from arXiv: 2607.10400 by Abhigya Verma, Amit Kumar Saha, Khyati Mahajan, Sagar Davasam, Sai Rajeswar Mudumba, Shruthan Radhakrishna, Vikas Yadav.

Figure 1
Figure 1. Figure 1: Landscape of benchmarks Top: benchmark comparison by average document length (pages) and average textual context (tokens). SynthDocBench occupies a unique region with both long multi-page context and high textual density. Bottom: comparison with existing chart and document VQA benchmarks. Prior benchmarks typically isolate either charts or long documents, whereas SYNTHDOCBENCH is designed to study their in… view at source ↗
Figure 2
Figure 2. Figure 2: Synthetic visual document generation pipeline. From a topic seed, the pipeline generates grounded report content, applies document-level visual styling, synthesizes visualizations, performs metadata and QA validation, and assembles the final HTML/PDF reports with a machine-readable QA manifest. Deterministic Key Information Extraction Query Generation Synthetic Reports Prose Infographics Autocorrection Str… view at source ↗
Figure 3
Figure 3. Figure 3: QA generation pipeline. The pipeline parses the generated report into structured evidence channels, extracts and synthesizes key information, and generates chart-reading, cross-modal, and multi-hop questions. A verification stage filters weak or malformed items before serializing the final QA output. or human labeling (full schema in Appendix L.3). Validation and assembly: Numeric values in both chart and … view at source ↗
Figure 4
Figure 4. Figure 4: Document composition statistics across 200 reports. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Hard failures: 109 questions where all six models score ≤ 3, broken down by error category and question subset (Ch. = chart-reading; Cx. = complex; XM. = cross￾modal). rankings on SYNTHDOCBENCH correlate with MMLongBench-Doc (Spearman ρ=0.657, Pearson r=0.683), supporting external validity of the benchmark ordering (Appendix C). PyMuPDF extracts page text (up to 60,000 chars) and GPT-4o answers without ima… view at source ↗
Figure 7
Figure 7. Figure 7: ACC (τ=6) by fine-grained question category. Rows are grouped by subset (Chart Reading, Cross-Modal, Complex Reasoning); columns are models ordered by overall ACC. The colour scale runs from red (0) to green (1). Full numerical values are in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Evaluation pipeline. Rendered PDFs are converted to page images at 144 DPI, grouped into concatenated 5-page strips, and supplied directly to candidate models at temperature 0. Candidate answers are then scored against deterministic reference answers by GPT-5 acting as the judge model J . B Benchmark Comparison [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of judge scores (0–10) per model. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Chart-reading ACC (τ=6, judge: GPT-5) by evidence position bucket (n=597, 200 reports). Questions bucketed by rel￾ative chart position p = k/K into equal thirds. Middle third is hardest for 4 of 6 models; Claude-Sonnet-4.5 steepest decline (−11.7 pp). AI & Technology Science & Environment Medicine & Health Economics & Policy Politics & Geopolitics Education 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Accuracy (score 6) 0… view at source ↗
Figure 12
Figure 12. Figure 12: ACC (τ=6) by question subset. Cross-modal questions are consistently hardest across all models, confirming modality alignment as the primary bottleneck. across all models, confirming modality alignment as the primary bottleneck in long-context document understanding. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Distribution of chart types (top 20 shown). The corpus covers 24 distinct types [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 7 linked inside Pith

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a. 12 Published as a conference paper at COLM 2026 Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Jialong Song, Peng Liu, et al. Qwen2.5-VL technica...

    Pith/arXiv arXiv 2026

  2. [2]

    M-longdoc: A benchmark for multimodal super-long document understanding and a retrieval-aware tuning framework

    Yew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song, Chaoqun Liu, Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super-long document understanding and a retrieval-aware tuning framework. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 9244–9261,

    2025

  3. [3]

    Zhen Li, Duan Li, Yukai Guo, Xinyuan Guo, Bowen Li, Lanxi Xiao, Shenyu Qiao, Jiashu Chen, Zijian Wu, Hui Zhang, Xinhuan Shu, and Shixia Liu

    URL https: //arxiv.org/abs/1711.00350. Zhen Li, Duan Li, Yukai Guo, Xinyuan Guo, Bowen Li, Lanxi Xiao, Shenyu Qiao, Jiashu Chen, Zijian Wu, Hui Zhang, Xinhuan Shu, and Shixia Liu. ChartGalaxy: A dataset for infographic chart understanding and generation.arXiv preprint arXiv:2505.18668,

    Pith/arXiv arXiv

  4. [4]

    Nelson F

    URLhttps://arxiv.org/abs/2505.18668. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts,

    arXiv

  5. [5]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279,

    2022

  6. [6]

    ChartQAPro: A more diverse and challenging benchmark for chart question answering

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aarya- man Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, et al. ChartQAPro: A more diverse and challenging benchmark for chart question answering. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 19123–19151,

    2025

  7. [7]

    OpenAI, Aaron Hurst, Adam Lerer, Adam Goucher, et al

    URLhttps://www.docvqa.org/. OpenAI, Aaron Hurst, Adam Lerer, Adam Goucher, et al. GPT-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  8. [8]

    OpenAI GPT-5 system card

    13 Published as a conference paper at COLM 2026 Aaditya Singh, Lawrence Chan, Ted Li, Amelia Glaese, et al. OpenAI GPT-5 system card. arXiv preprint arXiv:2601.03267,

    Pith/arXiv arXiv 2026

  9. [9]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025a. Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Gin...

    Pith/arXiv arXiv

  10. [10]

    Liyan Xu et al

    URLhttps://arxiv.org/abs/1502.05698. Liyan Xu et al. ChartMuseum: A benchmark for fine-grained chart understanding.arXiv preprint arXiv:2505.13444,

    Pith/arXiv arXiv

  11. [11]

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu

    URLhttps://arxiv.org/abs/2505.13444. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317–5327,

    arXiv

  12. [12]

    VisuLogic: A benchmark for evaluating visual reasoning in multi-step logic problems.arXiv preprint arXiv:2504.15279,

    Weiye Zhang et al. VisuLogic: A benchmark for evaluating visual reasoning in multi-step logic problems.arXiv preprint arXiv:2504.15279,

    Pith/arXiv arXiv

  13. [13]

    URL https://arxiv.org/abs/2504. 15279. Zifeng Zhu, Mengzhao Jia, Zhihan Zhang, Lang Li, and Meng Jiang. Multichartqa: Bench- marking vision-language models on multi-chart problems. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Pape...

    2025

  14. [14]

    A Evaluation Pipeline Diagram Figure 8 details the end-to-end evaluation pipeline used to assess all candidate models on SYNTHDOCBENCH. Rendered PDFs are rasterized to page images at 144 DPI, concatenated into 5-page vertical strips, and passed directly to each candidate vision-language model at temperature 0 (vision-only; no OCR or metadata). Each candid...

    2026

  15. [15]

    B Benchmark Comparison Table 7 situates SYNTHDOCBENCHrelative to the most closely related benchmarks and reports frontier model scores on each

    Candidate answers are then scored against deterministic reference answers by GPT-5 acting as the judge modelJ. B Benchmark Comparison Table 7 situates SYNTHDOCBENCHrelative to the most closely related benchmarks and reports frontier model scores on each. Despite strong numbers on DocVQA, ChartQA, and MMLongBench-Doc, coverage is sparse and inconsistent ac...

    2026

  16. [16]

    Table 9: Judge agreement stratified by question type on the full SYNTHDOCBENCH(200 reports, 1,788 questions)

    is itself evidence that SynthDocBench provides complementary diagnostic signal beyond existing benchmarks. Table 9: Judge agreement stratified by question type on the full SYNTHDOCBENCH(200 reports, 1,788 questions). r: Pearson correlation. w1: within-1-point agreement. ∆ACC: ACC(alt) − ACC(GPT-5) at τ=6. Positive ∆ACC means the alternative judge is more ...

    2026

  17. [17]

    17 Published as a conference paper at COLM 2026 0 2 4 6 8 10 Judge score (0

    2026

  18. [18]

    F Question Quality Control We apply three layers of quality assurance to the generated questions

    Claude-Sonnet-4.5 Gemini-3.1-Pro GPT-4o GPT-5.4 InternVL3-78B Qwen2.5-VL-7B Qwen3.5-VL-122B Qwen3-VL-235B ACC threshold (6) Figure 9: Distribution of judge scores (0–10) per model. F Question Quality Control We apply three layers of quality assurance to the generated questions. Numeric recomputation.For every question whose ground-truth answer is a single...

    2026

  19. [19]

    Best per row inbold

    Category Gemini-3.1 GPT-4o Claude-S-4.5 Qwen3-VL InternVL3 Qwen2.5-7B Chart Reading Value reading0.7840.271 0.382 0.648 0.437 0.015 Comparison0.7590.377 0.266 0.623 0.327 0.075 Trend / pattern 0.7340.7240.412 0.653 0.603 0.397 Cross-Modal Verify with chart0.6160.399 0.278 0.480 0.318 0.045 Integrate sources0.6920.298 0.260 0.545 0.293 0.030 Compare repr.0...

    arXiv 2026

  20. [20]

    Cross-modal questions are consistently hardest across all models, confirming modality alignment as the primary bottleneck

    0.35 0.34 0.25 0.76 0.79 0.63 0.46 0.36 0.34 0.42 0.46 0.39 0.46 0.40 0.30 0.16 0.01 0.07 0.71 0.69 0.56 0.64 0.61 0.50 Claude-Sonnet-4.5 Gemini-3.1-Pro GPT-4o GPT-5.4 InternVL3-78B Qwen2.5-VL-7B Qwen3.5-VL-122B Qwen3-VL-235B Figure 12: ACC (τ=6) by question subset. Cross-modal questions are consistently hardest across all models, confirming modality alig...

    2026

  21. [21]

    increasing trend,

    Each category belongs to one of the three question subsets and targets a distinct reasoning capability. The difficulty label L1–L5 is orthogonal to category: any category can appear at any difficulty level depending on the complexity of the evidence chain. Level Modality Operation Example L1: Direct lookup Chart / Table Read-offWhat is the value for X in ...

    2026

  22. [22]

    not found in the document

    27 Published as a conference paper at COLM 2026 Table 22: Error category definitions for the hard failure analysis (Section 5). Categories are mutually exclusive; each failure is assigned exactly one. Category Definition Visual Hallucination The model confidently reads a wrong value from the chart. The figure is located and the relevant element identified...

    2026

  23. [23]

    Each example shows (i) a zoomed- 29 Published as a conference paper at COLM 2026 in crop of the relevant evidence page, (ii) the question and ground-truth answer, and (iii) the annotated response from a representative subset of evaluated models, wheregreenmarks correct facts andredmarks hallucinated or wrong values. 30 Published as a conference paper at C...

    2026

  24. [24]

    not found

    All six evaluated models fail to localise and resolve the individual data point label, returning a uniform “not found” despite the chart being visually present. 33 Published as a conference paper at COLM 2026 Example 4 — Complex Multi-Hop L5Failure: Incomplete Retrieval Two of three evidence figures from p.5 of the gender equality document: Figure 3 (Repr...

    2026

  25. [25]

    Between 2013 and 2023, global child marriage rates declined from25%to19%

    and24new women to the House of Representatives. Between 2013 and 2023, global child marriage rates declined from25%to19%. During the 2017–2023 timeframe, the 50-point drop in the US Reproductive Rights Index (from75to25) contrasted with a 60-point increase in Ireland (from20to80) and a 45-point increase in Argentina (from30to75). GPT-4o 2/10: Following th...

    2013

  26. [26]

    reproductive rights index dropped by50 pointsfrom 2017 to 2023, during the same timeframe, Ireland’s index rose by25 pointsand Argentina’s by30 points

    While the U.S. reproductive rights index dropped by50 pointsfrom 2017 to 2023, during the same timeframe, Ireland’s index rose by25 pointsand Argentina’s by30 points. Claude-Sonnet-4.5 0/10: Not found in the document. Qwen3-VL-235B 3/10: Following the 1992 ‘Year of the Woman’, the number of women in the US Congress increased from 4to6. Global child marria...

    2017

  27. [27]

    InternVL3-78B 4/10: 34 Published as a conference paper at COLM 2026 Following the 1992 ‘Year of the Woman’ election, the number of women elected to the U.S

    The US reproductive rights index dropped by50 points, while Ireland saw a10-point increaseand Argentina experienced a10-point decreaseduring the same period. InternVL3-78B 4/10: 34 Published as a conference paper at COLM 2026 Following the 1992 ‘Year of the Woman’ election, the number of women elected to the U.S. House of Representatives increased by24, m...

    2026

  28. [28]

    reproductive rights index experienced a50-point drop, contrasting with Ireland and Argentina, which sawsignificant improvementsin their index scores during the 2017–2023 period

    The U.S. reproductive rights index experienced a50-point drop, contrasting with Ireland and Argentina, which sawsignificant improvementsin their index scores during the 2017–2023 period. The question requires six independent statistics from three charts on the same page. Gemini retrieves the Senate total and House count but uses wrong base scores for all ...

    2017