Pith. sign in

REVIEW 3 major objections 3 minor 76 references

FormStruct-Bench:A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recognition

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FormStruct-Bench shows table-form systems read content but miss structure, with fine-grained scores below 18 percent.

desk verdict Benchmark with a promising hierarchical protocol, but the headline 83.85% vs <18% gap is unverifiable without metric definitions and a human upper bound. read the letter →

arxiv 2608.10396 v1 pith:YYRJE2UO submitted 2026-08-11 cs.CV

Lujie Ban , Jiangtao Zhu , Yuanheng Yu , Jiasheng Shi , Chenhao Ma This is my paper · ORCID
classification cs.CV
keywords table-formdocumentstructurerecognitionhierarchicalbenchmarkunderstandingfine-grainedevaluationtemplateexpansiondiagnosticmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FormStruct-Bench is a benchmark for measuring whether table-form document systems recover not just the text but the multilevel structure—pages, schemas, and components—that organizes it. Using 70 annotated templates expanded into 7,000 verified instances, with a separate 1,100-instance human-reviewed test set, it evaluates 14 API-hosted and locally deployed systems plus two fine-tuned variants. The headline result is a large gap: the best document-level score is 83.85%, while the best fine-grained structural score is below 18%. The paper argues this shows current systems can read visible content but cannot reliably recover the hierarchy and regional layout needed for true table-form understanding, and that aggregate scores hide where such failures occur.

What carries the argument

The central object is FormStruct-Bench itself, a hierarchical benchmark built from 70 reusable templates expanded through a provenance-preserving Director–Artist–Verifier pipeline into 7,000 verified instances; all 1,100 template-disjoint test instances receive human review. The evaluation protocol uses five primary metrics and three structure-specific diagnostics at page, schema, and component levels, with slices over difficulty, structural constraints, and visual degradation. This design lets an aggregate score be decomposed into specific structural failure modes, which is what exposes the content-versus-structure gap.

What would settle it

Take a random sample of real-world table-form documents, annotate them with the same page, schema, and component protocol, and evaluate the same systems; if fine-grained structural scores approach document-level scores on that sample, the benchmark's template space—not the systems' capability—is what produces the reported gap.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art table-form document systems, measured on FormStruct-Bench, consistently extract visible content while failing at structural organization. The best system earns 83.85% at the document level, yet none exceeds 18% on fine-grained structural scores spanning page, schema, and component levels. This document-versus-structure gap is the discovery: it is not a single weak system but a systematic property across 14 systems and two SFT variants. The paper therefore contends that structural recognition, not content extraction, is the unsolved bottleneck for converting table-form documents into machine-processable records.

Load-bearing premise

The load-bearing premise is that the 70 templates, expanded into 7,000 instances, faithfully represent the variety of real-world table-form documents; if the templates are too narrow or artificial, the measured gap may be an artifact of the generator rather than a property of deployed systems.

Editorial extensions

If this is right

  • If the gap is real, improving table-form understanding requires targeting hierarchy and regional organization, not just OCR or content extraction.
  • Benchmark scores can now be traced to failure modes, so a system's high document-level score and low structural score identify exactly which components need work.
  • The template-disjoint, human-reviewed test set gives a reusable testbed for measuring progress on structural recognition.
  • The 14-system evaluation provides a baseline that future systems must be compared against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same diagnostic protocol could transfer to other structured documents such as invoices, receipts, and scientific tables to check whether the content-versus-structure gap generalizes.
  • Downstream applications like table question answering or database construction likely inherit the structural errors measured here, so fine-grained scores may predict end-task failures better than document-level scores.
  • Because the 70 templates are the source of all instances, testing on a broader corpus of real-world forms could narrow or widen the reported gap, clarifying whether it is inherent to current systems or amplified by template diversity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. FormStruct-Bench is introduced as a hierarchical benchmark for table-form document structure recognition. The authors describe a pipeline that annotates 70 reusable templates and expands them into 7,000 verified instances, with 1,100 template-disjoint test instances receiving additional human review. The protocol uses five primary metrics and three structure-specific diagnostics across page, schema, and component levels, with slices over difficulty, structural constraints, and visual degradation. The empirical study covers 14 API-hosted or locally deployable systems and two SFT variants; the best document-level score is 83.85%, while the best fine-grained structural score is below 18%, which the authors interpret as a pronounced gap between content reading and hierarchy recovery.

Significance. If the benchmark's claims are substantiated, FormStruct-Bench would fill a genuine gap: existing benchmarks focus on holistic output or conventional grids and do not localize structural failures. The provenance-preserving Director-Artist-Verifier pipeline and template-disjoint test set are methodical choices that mitigate annotation leakage and template memorization. The diagnostic orientation toward page, schema, and component levels is a useful contribution for failure analysis. However, the significance is conditional: the headline gap is only meaningful if the metrics are commensurable and if the template space is representative. The abstract alone does not establish these conditions, so the empirical conclusion must be regarded as preliminary.

major comments (3)
  1. [Abstract, results paragraph] The central claim that current systems recover visible content but not hierarchy rests on the gap between the best document-level score of 83.85% and the best fine-grained structural score below 18%. The abstract does not report any metric definitions, matching tolerances, or partial-credit rules, so readers cannot tell whether the two scores are commensurable: a fine-grained metric requiring exact hierarchical matches could be floor-limited even for a perfect structural model, whereas a loose token-level document metric would naturally be high. The absence of a human upper bound or a structure-oracle baseline makes it impossible to attribute the gap to model deficiency rather than scoring strictness. Please provide full metric definitions and at least one baseline that anchors the fine-grained scale.
  2. [Abstract, data construction] The 70 reusable templates expanded into 7,000 instances are asserted to be 'verified' and the 1,100 test instances are 'template-disjoint,' but the abstract gives no evidence that this template space is representative of real-world table-form documents. If the templates are narrow or synthetic, the observed gap may be an artifact of the generator rather than a property of deployed systems. The authors should describe how templates were selected or sampled, compare their statistics with real document collections, and report diversity measures across templates.
  3. [Abstract, evaluation protocol] Results for 14 systems plus two SFT variants are reported only as best scores. Without per-system variance, confidence intervals, or significance tests, the robustness of the 83.85% versus <18% gap is unclear, particularly because different systems may be optimized for different metrics. The paper should report error bars or permutation-based significance for the gap, and ideally a scatter of scores across systems.
minor comments (3)
  1. [Abstract, evaluation protocol] The abstract does not name the five primary metrics or the three diagnostics; including these names, even parenthetically, would help readers judge the protocol's scope.
  2. [Abstract, data construction] The terms 'Director-Artist-Verifier pipeline,' 'provenance-preserving,' and 'template-disjoint' are not defined; a one-sentence explanation in the abstract or a figure would improve accessibility.
  3. [Abstract, data construction] The abstract states that all 1,100 test instances received human review but does not report inter-annotator agreement; adding a Kappa or similar statistic would support the 'verified' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark reports measured scores rather than deriving predictions from fitted inputs.

full rationale

This is an abstract-only review. The paper introduces a benchmark, constructs ground truth via a template-expansion pipeline, and reports evaluation scores for existing systems. There is no derivation chain in which an input is defined in terms of an output, no fitted parameter is later presented as a prediction, and no load-bearing claim rests on a self-citation. The template-disjoint test set is explicitly designed to avoid the circularity of testing on templates used in generation. The only substantive concern, that the 83.85% versus <18% gap may reflect metric asymmetry rather than model deficiency, is a question of metric validity and commensurability, not circularity: the paper is not assuming the conclusion by construction. Without access to the full metric definitions, no specific equation or reduction can be shown to be equivalent to its inputs, so under the hard rules no circularity can be claimed.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are needed for a benchmark paper. The central claim depends on three domain assumptions about ground truth accuracy, template representativeness, and the validity of the template-disjoint split.

assumptions (3)
  • domain assumption The Director-Artist-Verifier pipeline and human review produce ground truth annotations that are correct and unambiguous.
    The value of every benchmark score depends on the quality of the labels. The abstract asserts verification and human review for the test set, but does not report inter-annotator agreement or quality audits.
  • domain assumption The 70 reusable templates span the practical space of table-form documents.
    The 7,000 instances are expansions of only 70 templates. If real-world layouts fall outside this template family, the reported performance gap will not transfer to actual deployment.
  • domain assumption The template-disjoint test set isolates model generalization rather than template memorization.
    The abstract states all 1,100 test instances are template-disjoint and human-reviewed, but the construction details needed to ensure this split is information-tight are not given in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FormStruct-Bench:A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recognition." pith.science (2026). https://pith.science/paper/YYRJE2UO

@misc{pith2026260810396,
  author       = {Pith},
  title        = {Pith review of: FormStruct-Bench:A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYRJE2UO}},
  note         = {Machine review of arXiv:2608.10396}
}
read the original abstract

Transforming table-form documents into machine-processable records requires recovering not only their visible content but also the multilevel structure that organizes it. However, existing benchmarks evaluate either holistic document outputs or conventional table grids, and their aggregate scores provide little insight into where structural failures occur. We introduce FormStruct-Bench, a hierarchical and diagnostic benchmark that evaluates table-form document structure recognition at both the document level and progressively finer component levels, allowing aggregate performance to be traced back to specific structural failure modes. To construct auditable ground truth at scale, we annotate 70 reusable templates and expand them into 7,000 verified instances through a provenance-preserving Director--Artist--Verifier pipeline; all 1,100 instances in the template-disjoint test set additionally receive human review. Our evaluation protocol uses five primary metrics and three structure-specific diagnostics across page, schema, and component levels, together with slices over difficulty, structural constraints, and visual degradation. Across 14 API-hosted and locally deployable systems plus two SFT variants, the best document-level score reaches 83.85%, whereas the best reported fine-grained structural score remains below 18%. These results reveal a pronounced gap between reading document content and recovering the hierarchy and regional organization required for reliable table-form understanding.

Figures

Figures reproduced from arXiv: 2608.10396 by the authors.

Figure 1
Figure 1. An example table-form document with its structural [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the FormStruct-Bench dataset construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Template distribution across coarse and fine do [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Template-disjoint split composition and distribu [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Performance across Difficulty L1–L4. in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Visual-degradation robustness: (a) degraded page [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Models fine-tuned on synthetic table-form docu [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Representative Rel-F1 errors after successful endpoint alignment. Red boxes and bottom insets identify the evaluated [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Evaluation sensitivity of API-hosted and local [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: illustrates how a table-form page is decomposed into nested semantic regions and fine-grained structural components. Colored boxes distinguish sections, table regions, keys, values, cells, and line-item groups, while orange connectors visualize group-level association…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 47 canonical work pages

  1. [1]

    Milan Aggarwal, Mausoom Sarkar, Hiresh Gupta, and Balaji Krishnamurthy

    work page

  2. [2]

    Anthropic. 2026. Claude Sonnet 5. Accessed: 2026-07-07. https://platform.claude. com/docs/en/about-claude/models/whats-new-sonnet-5

    work page 2026

  3. [3]

    Man- matha

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Man- matha. 2021. DocFormer: End-to-End Transformer for Document Understanding. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Montreal, QC, Canada, 973–983. doi:10.1109/ICCV48922.2021.00103

    arXiv 2021

  4. [4]

    Vasarhelyi

    Deniz Appelbaum, Alexander Kogan, and Miklos A. Vasarhelyi. 2017. Big Data and Analytics in the Modern Audit Engagement: Research Needs.Auditing: A Journal of Practice & Theory36, 4 (2017), 1–27. doi:10.2308/ajpt-51684

    doi:10.2308/ajpt-51684 2017

  5. [5]

    Udi Barzelay, Ophir Azulai, Inbar Shapira, Idan Friedman, Foad Abo Dahood, Madison Lee, and Abraham Daniels. 2026. VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents. arXiv:2603.15118 [cs.CV] doi:10.48550/ arXiv.2603.15118

    work page Pith review arXiv doi:10.48550/arxiv.2603.15118 2026

  6. [6]

    ByteDance Seed. 2026. Seed2.1 Model Card. Accessed: 2026-07-07. https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/ seed2.1/Seed2_1_Model_Card.pdf

    work page 2026

  7. [7]

    Tathagata Chakraborti, Vatche Isahagian, Rania Khalaf, Yasaman Khazaeni, Vinod Muthusamy, Yara Rizk, and Merve Unuvar. 2020. From Robotic Pro- cess Automation to Intelligent Process Automation. arXiv:2007.13257 [cs.AI] https://arxiv.org/abs/2007.13257

    arXiv 2020

  8. [8]

    Zewen Chi, Heyan Huang, Heng-Da Xu, Houjin Yu, Wanxuan Yin, and Xian-Ling Mao. 2019. Complicated Table Structure Recognition. arXiv:1908.04729 [cs.IR] https://arxiv.org/abs/1908.04729

    arXiv 2019

Show all 76 references
  1. [9]

    Common Crawl Foundation. 2024. Common Crawl Terms of Use. Terms for use of Common Crawl services and crawled content.. https://commoncrawl.org/terms- of-use

    2024

  2. [10]

    Common Crawl. 2026. Common Crawl Dataset. https://commoncrawl.org/

    2026

  3. [11]

    doc-analysis. 2022. XFUND: Repository License. CC BY-NC-SA 4.0 license statement.. https://github.com/doc-analysis/XFUND

    2022

  4. [12]

    Google AI for Developers. 2026. Gemini API Models. Accessed: 2026-07-07. https://ai.google.dev/gemini-api/docs/models

    2026

  5. [13]

    Ailin Huang, Chengyuan Yao, Chunrui Han, Fanqi Wan, Hangyu Guo, Haoran Lv, Hongyu Zhou, Jia Wang, Jian Zhou, Jianjian Sun, Jingcheng Hu, Kangheng Lin, Liang Zhao, Mitt Huang, Song Yuan, Wenwen Qu, Xiangfeng Wang, Yanlin Lai, Yingxiu Zhao, Yinmin Zhang, Yukang Shi, Yuyang Chen,...

  6. [14]

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. InProceed- ings of the 30th ACM International Conference on Multimedia. ACM, New York, NY, USA, 4083–4091. doi:10.1145/3503161.3548112

    2022

  7. [15]

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shjian Lu, and C. V. Jawahar. 2021. ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction. arXiv:2103.10213 [cs.AI] doi:10.1109/ICDAR.2019.00244

    2021 arXiv

  8. [16]

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents. arXiv:1905.13538 [cs.IR] https://arxiv.org/abs/1905.13538

    2019 arXiv

  9. [17]

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. OCR-free Document Understanding Transformer. arXiv:2111.15664 [cs.LG] https://arxiv.org/abs/2111.15664

    2022 arXiv

  10. [18]

    Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022. Form- Net: Structural Encoding beyond Sequential Modeling in Form Document Infor- mation Extraction. InProceedings of the 60th ...

    2022 doi

  11. [19]

    Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li

  12. [20]

    Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. 2020. DocBank: A Benchmark Dataset for Document Layout Analysis. arXiv:2006.01038 [cs.CL] https://arxiv.org/abs/2006.01038

    2020 arXiv

  13. [21]

    Jiefeng Ma, Yan Wang, Chenyu Liu, Jun Du, Yu Hu, Zhenrong Zhang, Pengfei Hu, Qing Wang, and Jianshu Zhang. 2024. SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding. InAdvances in Neural Information Processing Systems, Vol. 37. Cur...

    2024 doi

  14. [22]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. DocVQA: A Dataset for VQA on Document Images. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE, Online, 2200–2209. doi:10. 1109/WACV48630.2021.00225

    2021

  15. [23]

    Moonshot AI. 2026. Kimi. Accessed: 2026-07-07. https://www.kimi.com/

    2026

  16. [24]

    OpenAI. 2026. OpenAI API Models. Accessed: 2026-07-07. https://developers. openai.com/api/docs/models

    2026

  17. [25]

    Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. 2025. OmniDocBench: Benchmarking Diverse ...

    2025 arXiv

  18. [26]

    Nassar, and Peter Staar

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar. 2022. DocLayNet: A Large Human-Annotated Dataset for Document- Layout Segmentation. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USA...

    2022

  19. [27]

    Qwen Team. 2026. Qwen: Official Models and Chat Service. Accessed: 2026-07-07. https://qwen.ai/

    2026

  20. [28]

    Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5

    2026

  21. [29]

    Qwen Team. 2026. Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All. https://qwen.ai/blog?id=qwen3.6-35b-a3b

    2026

  22. [30]

    Josselin Somerville Roberts, Tony Lee, Chi Heem Wong, Michihiro Yasunaga, Yifan Mai, and Percy Liang. 2024. Image2Struct: Benchmarking Structure Ex- traction for Vision-Language Models. NeurIPS 2024. arXiv:2410.22456 [cs.CV] https://arxiv.org/abs/2410.22456

    2024 arXiv

  23. [31]

    Štěpán Šimsa, Milan Šulc, Michal Uřičář, Yash Patel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalický, Jiří Matas, Antoine Doucet, Mickaël Coustaty, and Dimosthenis Karatzas. 2023. DocILE Benchmark for Document Information Localization and Extraction. arXiv:2302.05658 [cs.CL] https:...

    2023 arXiv

  24. [32]

    Brandon Smock, Rohith Pesala, and Robin Abraham. 2021. PubTables- 1M: Towards comprehensive table extraction from unstructured documents. arXiv:2110.00061 [cs.LG] https://arxiv.org/abs/2110.00061

    2021 arXiv

  25. [33]

    Brandon Smock, Rohith Pesala, and Robin Abraham. 2023. GriTS: Grid table similarity metric for table structure recognition. arXiv:2203.12555 [cs.LG] https: //arxiv.org/abs/2203.12555

    2023 arXiv

  26. [34]

    Wil M. P. van der Aalst, Martin Bichler, and Armin Heinzl. 2018. Robotic Process Automation.Business & Information Systems Engineering60, 4 (2018), 269–272. doi:10.1007/s12599-018-0542-4

    2018 doi

  27. [35]

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. 2024. MinerU: An Open-Source Solution for Precise Document Content Extractio...

    2024 arXiv

  28. [36]

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Z...

    2025 doi arXiv

  29. [37]

    Yandi Wang, Libin Zhan, Ziwei Huang, Tiancheng Luo, Yuxuan Jiang, Wang Dong, Leilei Gan, and Jun Chen. 2026. From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding. In Proceedings of the 64th Annual Meeting of the Associati...

    2026

  30. [38]

    Haoran Wei, Yaofeng Sun, and Yukun Li. 2026. DeepSeek-OCR 2: Visual Causal Flow. arXiv:2601.20552 [cs.CV] doi:10.48550/arXiv.2601.20552 KDD ’27, August 01–05, 2027, San Jose, CA Ban et al

    2026 doi

  31. [39]

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. LayoutLM: Pre-training of Text and Layout for Document Image Understanding. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, USA,...

    2020

  32. [40]

    Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2021. LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding. arXiv:2104.08836 [cs.CL] https://arxiv. org/abs/2104.08836

    2021 arXiv

  33. [41]

    Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2022. XFUND: A Benchmark Dataset for Multilingual Vi- sually Rich Form Understanding. InFindings of the Association for Computational Linguistics: ACL 2022. Association for Compu...

    2022 doi

  34. [42]

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou

  35. [43]

    Z.ai. 2025. GLM-4.6V Model Card. Accessed: 2026-07-07. https://huggingface. co/zai-org/GLM-4.6V

    2025

  36. [44]

    Zelun Zhang, Hongen Liu, Suyin Liang, Yubo Zhang, Yiqing Xiang, Jiaxuan Liu, Ting Sun, Manhui Lin, Yue Zhang, Changda Zhou, Tingquan Gao, Cheng Cui, Yi Liu, Dianhai Yu, and Yanjun Ma. 2026. PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Regio...

    2026 arXiv

  37. [45]

    Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: data, model, and evaluation. arXiv:1911.10683 [cs.CV] https: //arxiv.org/abs/1911.10683

    2020 arXiv

  38. [46]

    InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing

    LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Un- derstanding. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing. Association for Computationa...

    doi

  39. [50]

    Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. PubLayNet: largest dataset ever for document layout analysis. arXiv:1908.07836 [cs.CL] https: //arxiv.org/abs/1908.07836 FormStruct-Bench : A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recogni...

    2019 arXiv

  40. [51]

    Return only strict JSON

    Task and output contract You are extracting the semantic answer tree and the minimal visible form structure needed for FormStruct-Bench evaluation. Return only strict JSON. Do not include markdown fences, prose, comments, or explanations. Do not include thinking tags such as <...

  41. [52]

    •Every object member must be a"key": valuepair

    JSON validity requirements: •The response must be parseable by a standard JSON parser. •Every object member must be a"key": valuepair. Do not put a bare string inside an object. •Escape quotation marks and line breaks inside strings. •Close every object and array that you open

  42. [53]

    answer": {

    Required top-level schema: { "answer": { "visible form title or top-level section": { "visible field label": "visible filled value", "nested visible section": { "visible field label": "visible filled value" } } }, "regions": [ {"id": "r1", "type": "title|section|field|value|te...

  43. [54]

    value":

    Answer rules: •Preserve visible labels as keys as closely as possible, including the original language/script. •Preserve nested section/field hierarchy. •Use strings for filled values. Use objects for nested sections and arrays for repeated line-item rows. •For selected checkb...

  44. [55]

    relations

    Structure rules: •Use pixel coordinates relative to the input image:[left, top, right, bottom]. •Include visible titles, section headers, field labels, filled value boxes/text areas, and checkbox/radio/input widgets as regions. •Each region must haveid,type,bbox, andtext. •Inc...

    2027

  45. [56]

    You will receive a list of semantic fields that need content for text boxes

    Task and input contract You are generating form-filling ACTIONS. You will receive a list of semantic fields that need content for text boxes. Each input item includesfield_key,data_type, andsemantic_key

  46. [57]

    •Output actions must use the samefield_keystring

    Field-key preservation rules •field_keyis the exact output key chosen by the planner. •Output actions must use the samefield_keystring. •Do not rename, translate, normalize, or deduplicatefield_key. • If the printed label is missing or unusable, the planner may fall back tosem...

  47. [58]

    •For each input item, output exactly one action in the same order

    Output format and action schema •Output only a JSON array of actions, with no extra text or markdown. •For each input item, output exactly one action in the same order. •Do not invent extra actions or checkbox actions. •Theaction_typemust matchdata_type: number fields usenumbe...

  48. [59]

    •Preservefield_keyas the output key, while usingsemantic_keyto infer the intended field meaning

    Semantic value generation rules •Use bothfield_keyandsemantic_keywhen generating content. •Preservefield_keyas the output key, while usingsemantic_keyto infer the intended field meaning. •Iffield_keyis ambiguous, generic, abbreviated, or not human-friendly, rely onsemantic_key...

  49. [60]

    Full-date fields use localized formats, while explicit date-component fields such as year, month, or day must contain only the corresponding component

    Localized date and partial-date rules Date values should follow the language and form style implied by field_key and semantic_key. Full-date fields use localized formats, while explicit date-component fields such as year, month, or day must contain only the corresponding compo...

  50. [61]

    •Do not copysemantic_keyas content unless it is genuinely the intended filled value

    Anti-copying and self-check rules •Do not copyfield_keyas the generated content. •Do not copysemantic_keyas content unless it is genuinely the intended filled value. •Generic option labels must be expanded into more specific plausible values. • Before returning the final JSON ...

  51. [62]

    field_key

    Output action schema [ { "field_key": "EXACT_FIELD_KEY_FROM_INPUT", "action_type": "write_text|number|date", "content": "GENERATED_FIELD_VALUE", "bbox": [x1, x2, y1, y2], "semantic_key": "EXACT_SEMANTIC_KEY_FROM_INPUT" } ] KDD ’27, August 01–05, 2027, San Jose, CA Ban et al. A...

    2027

  52. [63]

    The input image is a deterministic draft produced from structured form-filling actions

    Task and input contract You are editing an already-filled form draft image. The input image is a deterministic draft produced from structured form-filling actions. It already contains all required filled values, repeated field occurrences, checkbox selections, radio selections...

  53. [64]

    •Do not generate new semantic content

    Artist role and scope •Your task is to make the filled form look realistic and naturally completed. •Do not generate new semantic content. •Do not decide which fields should be filled. •Do not change the meaning, value, order, or location of any filled content. •Use the draft ...

  54. [65]

    •Do not add, remove, rewrite, paraphrase, translate, substitute, merge, or reorder any filled value

    Filled-content preservation rules •Preserve every filled value exactly as shown in the draft and as specified in the reference answer JSON. •Do not add, remove, rewrite, paraphrase, translate, substitute, merge, or reorder any filled value. •Preserve repeated field occurrences...

  55. [66]

    Template preservation rules •Treat all fixed printed template content as locked background. •Do not redraw, rewrite, translate, paraphrase, stylize, or alter printed labels, instructions, table lines, borders, logos, seals, stamps, URLs, organization names, footer text, or boi...

  56. [67]

    Visual naturalization rules •Only naturalize user-entered content. • Make the filled entries look like realistic form entries, such as handwritten text, typed text, mild pen-pressure variation, slight ink variation, natural baseline variation, and subtle scan noise. •Keep all ...

  57. [68]

    •Do not add new option marks

    Option-mark preservation rules •Preserve checkbox, radio, and circled-option states exactly as shown in the draft. •Do not add new option marks. •Do not remove existing option marks. •Do not change selected options into unselected options. •Do not change unselected options int...

  58. [69]

    •Before producing the final edited image, silently check that every filled value, repeated occurrence, and option state is preserved

    Critical self-check rules •The final image must still exactly match the reference answer JSON. •Before producing the final edited image, silently check that every filled value, repeated occurrence, and option state is preserved. •When visual realism conflicts with textual or p...

    2027

  59. [70]

    You will receive full form images plus a few same-class examples with their standard answer.json outputs

    Task and input contract You are an expert form-layout analysis and key-value extraction assistant. You will receive full form images plus a few same-class examples with their standard answer.json outputs. Each example contains one same-class form image and its standard structu...

  60. [71]

    •The extracted JSON will be compared against the original referenceanswer.json

    Validation role •Your output will be used for automatic field-level validation. •The extracted JSON will be compared against the original referenceanswer.json. • Missing fields, mismatched values, incorrect option states, or invalid structure may cause the target image to be r...

  61. [72]

    • Preserve the same field names, nesting levels, array structure, list order, option representation, and value style whenever the target form uses the same fields

    Few-shot schema learning rules •Use the example images and standardanswer.jsonfiles to learn the form class’s JSON hierarchy. • Preserve the same field names, nesting levels, array structure, list order, option representation, and value style whenever the target form uses the ...

  62. [73]

    •Traverse the entire image from the header or top row to the bottom

    Target extraction rules •Extract a complete structured key-value JSON object from the target image. •Traverse the entire image from the header or top row to the bottom. •Do not skip any visible region. •Treat label-like or title-like text as keys. •Match each key to its value ...

  63. [74]

    •Values may be strings, nested JSON objects, or arrays

    Value fidelity rules •Output every nested key-value relationship. •Values may be strings, nested JSON objects, or arrays. •Dates must be complete. •Preserve date ranges when present. •Preserve visible details such as numbers, emails, addresses, dates, punctuation, capitalizati...

  64. [75]

    •Do not infer selected options from row order, field defaults, or the few-shot examples

    Option-state extraction rules •For checkboxes, radio buttons, dropdowns, and option groups, output the option text actually selected in the target image. •Do not infer selected options from row order, field defaults, or the few-shot examples. •Do not output unselected options ...

  65. [76]

    •The response must start with{and end with}

    Output format rules •Return exactly one JSON object. •The response must start with{and end with}. •The JSON must be valid and fully closed. •Do not include explanations, Markdown, comments, confidence scores, code fences, error labels, or extra wrapper keys. •Return the finala...

    2027

  66. [2020]

    arXiv:1903.01949 [cs.CV] https://arxiv.org/abs/1903.01949

    TableBank: A Benchmark Dataset for Table Detection and Recognition. arXiv:1903.01949 [cs.CV] https://arxiv.org/abs/1903.01949

    1903 arXiv

  67. [2021]

    arXiv:2107.04396 [cs.CV] https://arxiv.org/abs/2107.04396

    Multi-Modal Association based Grouping for Form Structure Extraction. arXiv:2107.04396 [cs.CV] https://arxiv.org/abs/2107.04396

    arXiv

  68. [2026]

    arXiv:2601.09668 [cs.CV] doi:10.48550/ arXiv.2601.09668

    STEP3-VL-10B Technical Report. arXiv:2601.09668 [cs.CV] doi:10.48550/ arXiv.2601.09668

    doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.