Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Flagship multimodal models still fail at fine-grained layout of complex Bangla government forms, even when key-value extraction is strong.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 04:55 UTC pith:6TXN434E

load-bearing objection Solid first Bangla form benchmark with careful 26-class annotations and clear MLLM gaps; modest scale and MLLM-only scope are real but already scoped. the 3 major comments →

arxiv 2607.05614 v1 pith:6TXN434E submitted 2026-07-06 cs.CL cs.AIcs.CV

BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

Abu Tyeb Azad , Ishita Sur Apan , Fahim Ahmed , Sumaiya Karim Katha , Ezharuddin Jubaer , Armun Alam , Pranjal Kumar Nandi , Amin Ahsan Ali
show 3 more authors
Aman Chadha Md Mofijul Islam AKM Mahbubur Rahman
This is my paper
classification cs.CL cs.AIcs.CV
keywords document layout analysiskey information extractionBanglamultimodal large language modelsform understandinglow-resource NLPgovernment formsbenchmark dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces BaFCo, the first public benchmark built for complex Bangla form understanding. It collects 200 multi-page Bangladeshi government forms from many administrative domains and annotates them with a 26-class layout taxonomy (plus a 5-class coarse grouping) and key-value relationships. The authors then test current flagship multimodal models with zero-shot and chain-of-thought prompts under low and high reasoning budgets. The central finding is that these models remain weak at precise geometric localization of fine-grained form entities, while they perform substantially better at extracting the textual values once the keys are known. A sympathetic reader cares because government forms are how citizens interact with public services, and without reliable layout and extraction tools the benefits of document AI stay out of reach for a major low-resource language.

Core claim

Current off-the-shelf multimodal large language models cannot yet accurately localize highly granular form entities on complex Bangla government forms (best reported granular mAP@0.3 is 0.1177), whereas key-information extraction on the same documents reaches much higher F1 scores (up to 0.848). Layout difficulty is driven more by spatial granularity and hierarchical structure than by language itself.

What carries the argument

BaFCo: a curated set of 200 multi-page Bangladeshi government forms annotated with 26 fine-grained layout entity types (and a mapped 5-type coarse set), entity relationships, and 1,926 key-value pairs, evaluated under controlled zero-shot / chain-of-thought and low / high-reasoning regimes.

Load-bearing premise

The 200 carefully filtered public government forms and the 26-class taxonomy derived from them are assumed to be representative enough of real-world Bangla form diversity that the measured model gaps will generalize beyond this curated set.

What would settle it

A larger or differently sampled collection of Bangla forms (or a re-annotation under a coarser/finer taxonomy) on which the same flagship models achieve granular layout mAP comparable to their coarse or English results would falsify the claim that current MLLMs are systematically limited on fine-grained Bangla form layout.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Document AI pipelines for Bangla public services must still treat fine-grained layout analysis as an unsolved bottleneck rather than a solved precursor to extraction.
  • Coarse 5-class layouts are markedly easier for current models than 26-class taxonomies, so practical systems may need hierarchical or progressive localization.
  • Language itself is not the dominant barrier for layout detection; the same models show similar DLA scores on matched English forms, so layout failure modes are primarily geometric.
  • Key-information extraction can already be useful on Bangla forms even when full layout understanding remains unreliable.
  • Open-source models close much of the KIE gap but still lag proprietary leaders on precise bounding-box localization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same evaluation protocol could be applied to other low-resource South Asian scripts that share dense multi-column government-form conventions, testing whether the geometric bottleneck is script-general.
  • Because chain-of-thought and higher reasoning effort yield only mixed or negligible gains on layout, future gains are more likely to come from better visual-spatial grounding or layout-specific post-training than from longer textual reasoning.
  • The long-tailed entity distribution and hierarchical table structures in BaFCo make it a natural stress test for any future multimodal architecture that claims hierarchical region decomposition.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces BaFCo, a curated benchmark of 200 multi-page Bangladeshi government forms (316 pages) for Document Layout Analysis (DLA) and Key Information Extraction (KIE) in Bangla. It defines a 26-class fine-grained form-entity taxonomy plus a 5-class coarse mapping, with 16,382 entities, 8,771 relationships, and 1,926 key–value pairs, annotated by trained annotators with expert review (Cohen’s κ = 0.974). Flagship MLLMs (GPT-5.2, Gemini 3 Pro, Claude Opus 4.6, Qwen 3.6-Plus, Kimi K2.5) are evaluated zero-shot and with CoT under low/high reasoning. The central empirical claim is that current MLLMs remain weak at localizing highly granular form entities (best granular mAP@0.3 = 0.1177 for Gemini 3 Pro) while KIE is substantially stronger (Bangla F1 up to 0.848), with language effects small for DLA and larger for KIE.

Significance. If the measured gaps hold, BaFCo fills a clear low-resource gap: no prior public Bangla form benchmark supports fine-grained entities, key–value linking, multi-page government layouts, and dual DLA/KIE evaluation (Table 1). The dual granular/coarse taxonomy, English-control comparison (Tables 5–6), difficulty stratification, multi-domain coverage (15 domains), and public release of data and code make the result actionable for both model developers and the low-resource document-AI community. Annotation reliability (κ = 0.974) and standard external metrics (IoU-based mAP, NES) strengthen credibility. The work is primarily empirical rather than theoretical, but the resource and the documented MLLM failure modes (spatial shift, miscategorization, hallucination) are of practical significance for ECCV-style document understanding and for real-world Bangla public-service automation.

major comments (3)
  1. Sec. 4.2, Eqs. (1)–(2): invalid model outputs are discarded by the schema validator V(·) before scoring, but exclusion rates (by model, prompt, reasoning level, and entity set) are not reported. Because free-form MLLM bbox generation often fails format constraints, unreported filtering can inflate or deflate mAP relative to true layout competence and makes cross-model comparison harder to interpret. Please report the fraction of pages/predictions rejected and, if non-negligible, provide a secondary score that treats schema failures as empty predictions.
  2. Sec. 4.1 and Tables 4–6: the evaluation is restricted to off-the-shelf flagship MLLMs with no specialized document layout or KIE baseline (e.g., LayoutLMv3-style, DocLLM, or a strong OCR+rule pipeline) and no human upper bound. The authors justify the MLLM focus, but without at least one calibrated reference system it is difficult to separate “Bangla form hardness” from “MLLM bbox generation weakness.” A single specialized or OCR-based baseline on the same splits would substantially strengthen the claim that the observed gaps are task-level rather than interface-level.
  3. Sec. 3.1–3.2 and Limitations: the 200 forms are carefully filtered public government forms with a quality-over-quantity design. The headline claim about “current MLLMs’ ability in comprehending Bangla forms” is therefore scoped to this curated distribution. The paper already acknowledges limited scale; still, the main text should more explicitly bound generalization (e.g., private-sector, handwritten-heavy, or non-portal forms) so that the measured mAP/F1 numbers are not over-read as language-wide.
minor comments (6)
  1. Table 5: absolute counts of English DLA forms/pages used in the Bangla–English comparison are not stated as clearly as the KIE ratios in Sec. 3.2; please add them for reproducibility.
  2. Fig. 1 and Appendix figures: several qualitative panels are dense; larger crops or color-coded legends for predicted vs. ground-truth boxes would improve readability in print.
  3. Sec. 3.2 difficulty rubric: “easy/medium/hard” criteria are described narratively; a short checklist table (presence of tables, checkboxes, nested cells, etc.) would make the stratification fully reproducible.
  4. Throughout: minor consistency issues (e.g., “Gemini 3 Pro” vs “Gemini-3 Pro”, “Qwen 3.6 Plus” vs “Qwen 3.6-Plus”) should be normalized.
  5. Related Work: BaDLAD is correctly positioned as coarse-layout only; a one-sentence note on whether any of its government pages overlap BaFCo would help readers assess independence.
  6. Appendix D prompts: the full prompt templates are valuable; consider also releasing them as plain-text files in the public code repo for exact reproduction.

Circularity Check

0 steps flagged

No circularity: pure empirical benchmark with external standard metrics and no fitted or self-defined predictions.

full rationale

BaFCo is a dataset-and-evaluation paper. Its central claims are (1) the construction of a 200-form Bangla government-form corpus with a 26/5-class taxonomy and (2) measured zero-shot/CoT performance of off-the-shelf MLLMs under fixed IoU and NES metrics (Tabs. 4–6). No parameter is fitted to any subset of the data and then re-reported as a prediction; mAP, F1 and NES are computed against held-out human annotations by standard greedy matching. There is no uniqueness theorem, no self-citation that supplies a load-bearing premise, and no renaming of a known empirical pattern. The English-control comparison and the granular-vs-coarse ablation are independent experimental axes, not circular reductions. The paper is therefore self-contained against external benchmarks; circularity score is zero.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 1 invented entities

The paper is an empirical dataset + evaluation contribution. It inherits standard detection metrics and MLLM prompting practices; the only substantive modeling choices are the entity taxonomy and the difficulty rubric, both derived from inspection of the collected forms rather than free parameters fitted to performance numbers.

free parameters (1)
  • IoU matching thresholds τ ∈ {0.3, 0.5}
    Chosen by reference to Form-NLU and OmniDocBench rather than tuned on BaFCo; still a discrete design choice that affects absolute mAP values.
axioms (3)
  • standard math Standard object-detection matching (greedy IoU assignment) and classification metrics (precision/recall/F1/mAP) correctly quantify layout quality.
    Adopted without modification from COCO-style and prior form-understanding literature (Sec. 4.1).
  • domain assumption The 26 granular (and 5 coarse) entity types exhaustively and consistently cover the structural and functional elements of Bangladeshi government forms.
    Taxonomy iteratively refined via pilot annotations (Sec. 3.3); treated as ground truth for all subsequent evaluation.
  • domain assumption Zero-shot and CoT prompting of flagship MLLMs, with low/high reasoning_effort, is a fair and informative evaluation regime for document understanding.
    Explicit methodological choice justified by API accessibility and scarcity of Bangla training data (Sec. 4.1).
invented entities (1)
  • 26-class fine-grained form-entity taxonomy (plus 5-class coarse mapping) no independent evidence
    purpose: Capture nested keys/values, table sub-structure, signatures, checkboxes, etc., that coarser schemes miss.
    Defined by the authors from analysis of the collected forms; no external independent validation beyond inter-annotator agreement.

pith-pipeline@v1.1.0-grok45 · 26296 in / 2511 out tokens · 32852 ms · 2026-07-11T04:55:54.085353+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension." pith.science (2026). https://pith.science/paper/6TXN434E

@misc{pith2026260705614,
  author       = {Pith},
  title        = {Pith review of: BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TXN434E}},
  note         = {Machine review of arXiv:2607.05614}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real-world, human-centric applications. However, this adoption is limited for low-resource languages such as Bangla due to the scarcity of high-quality annotated data. To address this gap, we introduce BaFCo, a benchmark dataset for Bangla form comprehension with a focus on Document Layout Analysis (DLA) and Key Information Extraction (KIE). BaFCo curates 200 multi-page complex Bangladeshi government forms, sourced from across diverse sectors including agriculture, education, banking, and land management. To accurately capture the structural and contextual complexity of these forms, we define a fine-grained annotation schema comprising 26 types of form entities, along with a separate coarse form entity set consisting of 5 types. We evaluate the latest MLLMs from the ChatGPT, Gemini, Claude, Qwen, and Kimi series using zero-shot and chain-of-thought prompts under both low and high reasoning setups. Our results reveal limitations in current MLLMs' ability in comprehending Bangla forms, particularly in accurately localizing highly granular form entities. Our dataset and code is available at: https://huggingface.co/datasets/Mausul/bafco

Figures

Figures reproduced from arXiv: 2607.05614 by Abu Tyeb Azad, AKM Mahbubur Rahman, Aman Chadha, Amin Ahsan Ali, Armun Alam, Ezharuddin Jubaer, Fahim Ahmed, Ishita Sur Apan, Md Mofijul Islam, Pranjal Kumar Nandi, Sumaiya Karim Katha.

Figure 1
Figure 1. Figure 1: Overview of BaFCo data diversity. (a) Examples of annotated forms across three difficulty levels: easy, medium, and hard (see Sec. 3.2); (b) BaFCo supports detailed layout annotations with 26 entity types, including titles, form key-value pairs, and tables; (c) It contains links between related entities through three relationship types: key-to-key, key-to-value, and value-to-value; (d) Examples of form var… view at source ↗
Figure 2
Figure 2. Figure 2: visualizes the ground-truth entity distribution under the coarse (5-class) and granular (26-class) label sets for DLA, with exact counts and percentages listed in Tab. 7. The distribution is strongly long-tailed, reflecting the natural composition of real-world Bangladeshi forms. At the coarse level, Fields domi￾nate (64.6%), followed by Table (27.3%), with Others (3.9%), Headings (3.8%), and Image (0.4%) … view at source ↗
Figure 3
Figure 3. Figure 3: shows one representative form per difficulty tier. The Easy example con￾tains only key–value fields and headings, with no tables or checkboxes. The Medium example contains a columns-only table alongside key–value fields. The Hard example contains a full row–column table with multiple pre-defined rows, characteristic of the densest forms in the dataset. (a) Easy (b) Medium (c) Hard [PITH_FULL_IMAGE:figures… view at source ↗
Figure 4
Figure 4. Figure 4: Failure and success modes of MLLM predictions for granular form entities [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Failure and success modes of MLLM predictions for coarse form entities [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Failure and success modes of MLLM predictions for KIE Task [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages

  1. [1]

    ACM Comput

    Binmakhashen, G.M., Mahmoud, S.A.: Document layout analysis: A comprehen- sive survey. ACM Comput. Surv.52(6) (Oct 2019).https://doi.org/10.1145/ 3355610

    2019

  2. [2]

    In: Proceed- ings of the 9th IAPR International Workshop on Document Analysis Systems

    Bukhari, S.S., Al Azawi, M.I.A., Shafait, F., Breuel, T.M.: Document image seg- mentation using discriminative learning over connected components. In: Proceed- ings of the 9th IAPR International Workshop on Document Analysis Systems. pp. 183–190. DAS ’10, Association for Computing Machinery, New York, NY, USA (2010).https://doi.org/10.1145/1815330.1815354

    work page doi:10.1145/1815330.1815354 2010

  3. [3]

    In: Proceedings of the 30th ACM International Conference on Multimedia

    Cao, H., Li, X., Ma, J., Jiang, D., Guo, A., Hu, Y., Liu, H., Liu, Y., Ren, B.: Query-driven generative network for document information extraction in the wild. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4261–4271. MM ’22, Association for Computing Machinery, New York, NY, USA (2022).https://doi.org/10.1145/3503161.3547877

    doi:10.1145/3503161.3547877 2022

  4. [4]

    In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

    Cao, H., Ma, J., Guo, A., Hu, Y., Liu, H., Jiang, D., Liu, Y., Ren, B.: GMN: Generative multi-modal network for practical document information extraction. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 3768–3778. Association for Computational Linguisti...

    work page doi:10.18653/v1/2022.naacl-main.276 2022

  5. [5]

    In: Bouamor, H., Pino, J., Bali, K

    Cao, P., Wang, Y., Zhang, Q., Meng, Z.: GenKIE: Robust generative multimodal document key information extraction. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 14702–14713. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.findings-emnlp.979

    work page doi:10.18653/v1/2023.findings-emnlp.979 2023

  6. [6]

    arXiv preprint arXiv:2405.21015 (2024)

    Cottier, B., Rahman, R., Fattorini, L., Maslej, N., Besiroglu, T., Owen, D.: The rising costs of training frontier ai models. arXiv preprint arXiv:2405.21015 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval (SIGIR ’23)

    Ding, Y., Long, S., Huang, J., Ren, K., Luo, X., Chung, H., Han, S.C.: Form-nlu: Dataset for the form natural language understanding. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval (SIGIR ’23). pp. 2807–2816. Association for Computing Machinery, Taipei, Taiwan (2023).https://doi.org/10...

    doi:10.1145/3539618.3591886 2023

  8. [8]

    arXiv preprint arXiv:2507.09861 (2025)

    Ding,Y.,Luo,S.,Dai,Y.,Jiang,Y.,Li,Z.,Martin,G.,Peng,Y.:Asurveyonmllm- based visually rich document understanding: Methods, challenges, and emerging trends. arXiv preprint arXiv:2507.09861 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    In: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Duan, Y., Chen, Z., Hu, Y., Wang, W., Ye, S., Shi, B., Lu, L., Hou, Q., Lu, T., Li, H., Dai, J., Wang, W.: Docopilot: Improving multimodal models for document- level understanding. In: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4026–4037 (2025).https://doi.org/10.1109/ CVPR52734.2025.00381

    arXiv 2025

  10. [10]

    In: Proceedings of the 2024 Joint International Confer- ence on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024)

    Fujitake,M.:LayoutLLM:Largelanguagemodelinstructiontuningforvisuallyrich document understanding. In: Proceedings of the 2024 Joint International Confer- ence on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024). pp. 10219–10224. ELRA and ICCL, Torino, Italia (May 2024), https://aclanthology.org/2024.lrec-main.892/

    2024

  11. [11]

    In: Proceedings of 3rd International Conference on Document Analysis and Recognition

    Ha, J., Haralick, R., Phillips, I.: Recursive x-y cut using bounding boxes of con- nected components. In: Proceedings of 3rd International Conference on Document Analysis and Recognition. vol. 2, pp. 952–955 vol.2 (1995).https://doi.org/10. 1109/ICDAR.1995.602059 16 A. Azad et al

    work page arXiv 1995

  12. [12]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Hong, T., Kim, D., Ji, M., Hwang, W., Nam, D., Park, S.: Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 10767–10775 (2022).https://doi.org/https://doi.org/10.1609/ aaai.v36i10.21322

    2022

  13. [13]

    In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

    Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Zhang, J., Jin, Q., Huang, F., Zhou, J.: mPLUG-DocOwl 1.5: Unified structure learning for OCR-free doc- ument understanding. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Find- ings of the Association for Computational Linguistics: EMNLP 2024. pp. 3096–

    2024

  14. [14]

    https://doi.org/10.18653/v1/2024.findings-emnlp.175

    Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10.18653/v1/2024.findings-emnlp.175

    work page doi:10.18653/v1/2024.findings-emnlp.175 2024

  15. [15]

    In: Proceedings of the 30th ACM International Conference on Multimedia

    Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: Layoutlmv3: Pre-training for docu- ment ai with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4083–4091. MM ’22, Association for Computing Machinery, New York, NY, USA (2022).https://doi.org/10.1145/ 3503161.3548112

    work page arXiv 2022

  16. [16]

    In: 2019 International Conference on Document Analysis and Recognition (ICDAR)

    Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., Jawahar, C.V.: Ic- dar2019 competition on scanned receipt ocr and information extraction. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1516–1520 (2019).https://doi.org/10.1109/ICDAR.2019.00244

    doi:10.1109/icdar.2019.00244 2019

  17. [17]

    In: 2019 International Conference on Doc- ument Analysis and Recognition Workshops (ICDARW)

    Jaume, G., Kemal Ekenel, H., Thiran, J.P.: Funsd: A dataset for form under- standing in noisy scanned documents. In: 2019 International Conference on Doc- ument Analysis and Recognition Workshops (ICDARW). vol. 2, pp. 1–6 (2019). https://doi.org/10.1109/ICDARW.2019.10029

    doi:10.1109/icdarw.2019.10029 2019

  18. [18]

    In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05)

    Journet, N., Eglin, V., Ramel, J., Mullot, R.: Text/graphic labelling of ancient printed documents. In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05). pp. 1010–1014 Vol. 2 (2005).https://doi.org/10. 1109/ICDAR.2005.235

    2005

  19. [19]

    In: Scott, D., Bel, N., Zong, C

    Li,M.,Xu,Y.,Cui,L.,Huang,S.,Wei,F.,Li,Z.,Zhou,M.:DocBank:Abenchmark dataset for document layout analysis. In: Scott, D., Bel, N., Zong, C. (eds.) Pro- ceedings of the 28th International Conference on Computational Linguistics. pp. 949–960. International Committee on Computational Linguistics, Barcelona, Spain (Online) (Dec 2020).https://doi.org/10.18653/v...

    work page doi:10.18653/v1/2020.coling-main.82 2020

  20. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Liao, W., Wang, J., Li, H., Wang, C., Huang, J., Jin, L.: Doclayllm: An efficient multi-modal extension of large language models for text-rich document under- standing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4038–4049 (June 2025)

    2025

  21. [21]

    In: European conference on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)

    2014

  22. [22]

    In: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Ouyang, L., Qu, Y., Zhou, H., Zhu, J., Zhang, R., Lin, Q., Wang, B., Zhao, Z., Jiang, M., Zhao, X., Shi, J., Wu, F., Chu, P., Liu, M., Li, Z., Xu, C., Zhang, B., Shi, B., Tu, Z., He, C.: Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp...

    doi:10.1109/cvpr52734.2025.02313 2025

  23. [23]

    In: Workshop on Document Intelligence at NeurIPS 2019 (2019) BaFCo Benchmark 17

    Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., Lee, H.: Cord: a consolidated receipt dataset for post-ocr parsing. In: Workshop on Document Intelligence at NeurIPS 2019 (2019) BaFCo Benchmark 17

    2019

  24. [24]

    In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    Pfitzmann, B., Auer, C., Dolfi, M., Nassar, A.S., Staar, P.: Doclaynet: A large human-annotated dataset for document-layout segmentation. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 3743–3751. KDD ’22, Association for Computing Machinery, New York, NY, USA (2022).https://doi.org/10.1145/3534678.3539043

    doi:10.1145/3534678.3539043 2022

  25. [25]

    In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V

    Poria, S., Huang, X.: Bhaasha, bh¯ as.¯ a, zaban: A survey for low-resourced lan- guages in South Asia – current stage and challenges. In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V. (eds.) Findings of the Association for Compu- tational Linguistics: EMNLP 2025. pp. 1386–1406. Association for Computational Linguistics, Suzhou, China (Nov 2...

    doi:10.18653/v1/2025 2025

  26. [26]

    In: 2023 10th International Conference on Wireless Networks and Mobile Commu- nications (WINCOM)

    Sassioui, A., Benouini, R., El Ouargui, Y., El Kamili, M., Chergui, M., Ouzzif, M.: Visually-rich document understanding: Concepts, taxonomy and challenges. In: 2023 10th International Conference on Wireless Networks and Mobile Commu- nications (WINCOM). pp. 1–7 (2023).https://doi.org/10.1109/WINCOM59760. 2023.10322990

    work page doi:10.1109/wincom59760 2023

  27. [27]

    In: Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San José, CA, USA, August 21-26, 2023, Proceedings, Part I

    Shihab, M.I.H., Hasan, M.R., Emon, M.R., Hossen, S.M., Ansary, M.N., Ahmed, I., Rakib, F.R., Dhruvo, S.E., Dip, S.S., Pavel, A.H., Meghla, M.H., Haque, M.R., Chowdhury, S.S., Sadeque, F., Reasat, T., Humayun, A.I., Sushmit, A.: Badlad: A large multi-domain bengali document layout analysis dataset. In: Document Analysis and Recognition - ICDAR 2023: 17th I...

    work page doi:10.1007/978-3-031-41676-7_19 2023

  28. [28]

    In: Document Analysis and Recognition - ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part I

    Stanisławek, T., Graliński, F., Wróblewska, A., Lipiński, D., Kaliska, A., Rosalska, P., Topolski, B., Biecek, P.: Kleister: Key information extraction datasets involving long documents with complex layouts. In: Document Analysis and Recognition - ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part I. ...

    work page doi:10.1007/978-3-030-86549-8_36 2021

  29. [29]

    United Nations (2024), https://www.un-ilibrary.org/content/books/9789211067286#overview

    United Nations Department of Economic and Social Affairs: United Nations E- Government Survey 2024: Accelerating Digital Transformation for Sustainable De- velopment - With the addendum on Artificial Intelligence. United Nations (2024), https://www.un-ilibrary.org/content/books/9789211067286#overview

    arXiv 2024

  30. [30]

    In: Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San José, CA, USA, August 21- 26, 2023, Proceedings, Part II

    Šimsa, v., Šulc, M., Uřičář, M., Patel, Y., Hamdi, A., Kocián, M., Skalický, M., Matas, J., Doucet, A., Coustaty, M., Karatzas, D.: Docile benchmark for document information localization and extraction. In: Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San José, CA, USA, August 21- 26, 2023, Proceedings, Part II. pp. 147–1...

    work page doi:10.1007/978-3-031-41679-8_9 2023

  31. [31]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Wang, D., Raman, N., Sibue, M., Ma, Z., Babkin, P., Kaur, S., Pei, Y., Nour- bakhsh, A., Liu, X.: DocLLM: A layout-aware generative language model for mul- timodal document understanding. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8529–

  32. [32]

    https://doi.org/10.18653/v1/2024.acl-long.463

    Association for Computational Linguistics, Bangkok, Thailand (Aug 2024). https://doi.org/10.18653/v1/2024.acl-long.463

    doi:10.18653/v1/2024.acl-long.463 2024

  33. [33]

    In: Chen, C.C., Moreno-Sandoval, A., Huang, J., Xie, Q., Ananiadou, S., Chen, H.H

    Wang, D., Zmigrod, R., Sibue, M.J., Pei, Y., Babkin, P., Brugere, I., Liu, X., Navarro, N., Papadimitriou, A., Watson, W., Ma, Z., Nourbakhsh, A., Shah, S.: BuDDIE: A business document dataset for multi-task information extraction. In: Chen, C.C., Moreno-Sandoval, A., Huang, J., Xie, Q., Ananiadou, S., Chen, H.H. (eds.) Proceedings of the Joint Workshop o...

    2025

  34. [34]

    In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Wang, J., Jin, L., Ding, K.: LiLT: A simple yet effective language-independent layout transformer for structured document understanding. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (May 2022).https: //doi.org/10.18653/v1/2022.acl-long.534

    work page doi:10.18653/v1/2022.acl-long.534 2022

  35. [35]

    In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari,C.,Niu,G.,Sabato,S.(eds.)Proceedingsofthe39thInternationalCon- ference on Machine Learning

    Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari,C.,Niu,G.,Sabato,S.(eds.)Proceedingsofthe39thInternationalCon- ference on Machine Learning. Proceeding...

    2022

  36. [36]

    arXiv preprint arXiv:2109.01652 (2021)

    Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., Le, Q.V.: Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)

    Pith/arXiv arXiv 2021

  37. [37]

    NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models.In:Proceedingsofthe36thInternationalConferenceonNeuralInformation Processing Systems. NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022)

    2022

  38. [38]

    Wu, C.C., Chou, C.H., Chang, F.: A machine-learning approach for analyzing documentlayoutstructureswithtworeadingorders.PatternRecogn.41(10),3200– 3213 (Oct 2008).https://doi.org/10.1016/j.patcog.2008.03.014

    work page doi:10.1016/j.patcog.2008.03.014 2008

  39. [39]

    In: 2024 IEEE Inter- national Symposium on Workload Characterization (IISWC)

    Xia, Y., Kim, J., Chen, Y., Ye, H., Kundu, S., Hao, C.C., Talati, N.: Understanding the performance and estimating the cost of llm fine-tuning. In: 2024 IEEE Inter- national Symposium on Workload Characterization (IISWC). pp. 210–223 (2024)

    2024

  40. [40]

    In: Muresan, S., Nakov, P., Villavicencio, A

    Xu, Y., Lv, T., Cui, L., Wang, G., Lu, Y., Florencio, D., Zhang, C., Wei, F.: XFUND: A benchmark dataset for multilingual visually rich form understanding. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Findings of the Association for Computational Linguistics: ACL 2022. pp. 3214–3224. Association for Computa- tional Linguistics, Dublin, Ireland (Ma...

    doi:10.18653/v1/ 2022

  41. [41]

    Zhang, J., Yang, W., Lai, S., Xie, Z., Jin, L.: Dockylin: a large multimodal model for visual document understanding with efficient visual slimming. In: Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Sym- posium on Educational Adva...

    work page doi:10.1609/aaai.v39i9.33076 2025

  42. [42]

    In: 2019 International Conference on Document Analysis and Recognition (ICDAR)

    Zhong, X., Tang, J., Jimeno Yepes, A.: Publaynet: Largest dataset ever for docu- ment layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1015–1022 (2019).https://doi.org/10.1109/ICDAR. 2019.00166 BaFCo Benchmark 19 Appendix A Form Entity Distribution Fig. 2 visualizes the ground-truth entity distribution u...

    doi:10.1109/icdar 2019

  43. [43]

    Localize every visible field on the page

  44. [44]

    Classify each field into one of the given form field categories

  45. [45]

    id": "<unique_id>

    Return bounding boxes [X, Y, W, H], where (X, Y) is the top−left corner, W is width, and H is height. OUTPUT FORMAT (exactly as shown): [ { "id": "<unique_id>", "reasoning": <brief 1−line non−generic reasoning on choice of class>, "class": "<category_name>", "bbox": <Length 4 list [X, Y, W, H] where (X, Y) represents left−upper corner of bounding box and ...

  46. [46]

    An image of a document

  47. [47]

    −Extract the exact visible value from the document

    A question in the format: What is the value of <key_text>? Your task: −Locate the field corresponding to <key_text>. −Extract the exact visible value from the document. −If multiple possible values are found, return the one closest to the key text in the document layout. Output rules: −Return the answer strictly inside these tags: <answer>the exact extrac...