Pith. sign in

REVIEW 3 major objections 4 minor 172 references

XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read XL-DocBench is a fully human-verified benchmark for extra-long document understanding, and it shows current AI systems still struggle: the best pipeline reaches 44% accuracy on documents up to 2,303 pages.

desk verdict A valuable benchmark with a carefully designed pipeline, but the unvalidated no-context filter and missing release/agreement metrics make the 'fully human-verified' claim unsupported. read the letter →

arxiv 2608.00036 v1 pith:3WGXY3OA submitted 2026-07-21 cs.CL cs.AI

Hongchen Wei , Yuanzhe Wang , Bei Liu , Yifan Yang , Qi Dai , Ruichun Ma , Kai Qiu , Yunsheng Li
show 4 more authors
Dongdong Chen Chong Luo Zhenzhong Chen Baining Guo
This is my paper · ORCID
classification cs.CLcs.AI
keywords long-documentunderstandingbenchmarkevidencegroundingretrieval-augmentedgenerationmultimodaldocumentQAunanswerablequestionsreasoningtaxonomyhumanverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces XL-DocBench, a benchmark of 1,519 expert-verified questions over professional documents up to 2,303 pages, designed to test whether language models can find and use evidence scattered across hundreds of pages. The authors' central claim is that long-context capability is not the only bottleneck: even the strongest evaluated pipeline answers only 44.0% of questions correctly, and failures concentrate in set-tracking (ranking, coverage, set difference) and knowing when to abstain. The benchmark pairs each question with page-level evidence annotations, one of twelve reasoning labels, and a typed verification rule, so a system's mistakes can be attributed to retrieval, evidence use, or rule-following rather than a single score. The stated purpose is to give future long-document QA research a diagnostic instrument suited to compliance, clinical, financial, and engineering workflows where unsupported answers are costly.

What carries the argument

The benchmark itself is the central object: a tree-guided synthesis pipeline that organizes each PDF into a hierarchy of sections and pages, uses a leave-one-branch-out test to ensure questions cannot be answered from a single chapter or section, then relies on 194 human experts to verify every retained question and annotate exact evidence pages and quotes. Each example carries a typed verification rule ('numeric tolerance,' 'choice exact match,' 'None-answer') and one of twelve reasoning labels, enabling failures to be attributed to specific reasoning steps rather than a single aggregate score.

What would settle it

Take a random sample of, say, 200 XL-DocBench questions and ask a strong language model to answer them without any document access. If it answers significantly above the expected floor (or if a re-verification by a second independent team disagrees with the released labels on more than a small fraction), the benchmark's claim to measure document understanding rather than world-knowledge leakage would be falsified.

Watch

Extended reading notes

Core claim

XL-DocBench establishes that current document-understanding systems, including 1M-token one-shot readers and retrieval-augmented agents, still fail on extra-long professional documents: the strongest pipeline reaches 44.0% overall accuracy, and no one-shot model exceeds 39.8%. The benchmark goes beyond page-level lookup: 72.6% of its 1,519 retained questions require multiple evidence pages, 36.6% require tables, charts, or figures, 10.9% require cross-document evidence, and 14.4% are unanswerable. Its diagnostic labels reveal that the binding constraints are not raw context length but the ability to track sets of evidence and to abstain when support is missing.

Load-bearing premise

The benchmark's diagnostic claim rests on the assumption that every retained question is genuinely unanswerable without the document and that the human-verified answers and evidence pages are correct—but no accuracy or agreement statistics are reported for the no-context filter or the human verifiers.

Editorial extensions

If this is right

  • If the benchmark's diagnostic claim holds, future work can attribute failures to retrieval, evidence use, or rule-following instead of treating a leaderboard score as the outcome.
  • Systems that cannot handle multi-page evidence, set-tracking, or abstention will be exposed even if they perform well on shorter single-page benchmarks.
  • The 44% ceiling suggests current long-context and agentic systems are not yet reliable for professional workflows where decisions must be traceable to specific pages.
  • Page-level evidence annotations and typed rules provide a template for building harder, more inspectable document-understanding benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's usefulness depends on the no-context filter being accurate; if leaked questions survive, scores could be inflated by world knowledge rather than document understanding. A second, independent no-context audit of a sample would test this directly.
  • The paper reports no inter-annotator agreement, so the reliability of the human verification step is unquantified; a reproducibility study with a second annotation team on a subset would strengthen the foundation.
  • The cross-document subset is small (165 questions) and the paper notes page-level statistics understate true cross-document scope; future work could expand this slice to test multi-document aggregation more robustly.
  • Because the benchmark is built on public professional documents, models pretrained on those documents may have an advantage; the diagnostic value could degrade over time as training data includes these sources.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces XL-DocBench, a proposed benchmark for extra-long, evidence-grounded document understanding, built from 1,519 human-verified questions over professional documents up to about 2,000--2,300 pages across six domains, with twelve reasoning types, expert-annotated evidence pages, typed verification rules, and diagnostic subsets for multimodal, cross-document, and unanswerable questions. The construction pipeline is tree-guided model-assisted synthesis with automatic artifact filters followed by verification by 194 human experts. The paper reports evaluations of one-shot models and agent pipelines, finding that the best system reaches 44.0% overall accuracy and that set tracking and abstention are the main failure modes. The central claims are that the benchmark is fully human-verified and that its page-level evidence and reasoning labels allow system failures to be attributed to retrieval, evidence use, or rule following.

Significance. If the central claims hold, XL-DocBench would fill a real gap: existing document-understanding benchmarks are mostly single-page, short multi-page, or text-only, while professional workflows require combining evidence across hundreds of pages, multiple documents, tables/charts, and explicit abstention. The paper's diagnostic breakdowns by reasoning type, evidence span, modality, and document scope, together with deterministic scoring and detailed failure case studies, are valuable. The construction pipeline is described in unusual detail, and the authors are explicit that LLM outputs are provisional and that human verification is the final quality gate. These are genuine strengths. However, the benchmark's diagnostic value depends on the final questions being unanswerable without the documents, and the only guard aimed at that requirement -- the no-context filter -- is not empirically validated. The dataset is also not currently released, so the human-verified labels and the proposed diagnostic breakdowns cannot be independently checked.

major comments (3)
  1. [§3.1, Appendix A; §3.3] The no-context filter is load-bearing for the central diagnostic claim, and it is unvalidated. The paper states it is the largest single rejection source and that it rejects candidates answerable from world knowledge, metadata, or wording leakage, but no judge model, threshold, or accuracy/precision/recall/agreement is reported. Human verification (§3.3) is performed with full document access and is not described as independently re-testing no-context answerability; a verifier who sees the source document can easily miss that a question is also answerable from parametric knowledge. Leaked questions would confound the reasoning-type and evidence-page breakdowns, because systems could score on world knowledge rather than document understanding. The manuscript needs an empirical no-context control on the final 1,519 examples (e.g., answer the question with no document content) and a validat
  2. [§1 (project homepage); Appendix D] The dataset, code, and even a valid project URL are not available. The paper says 'Project homepage:/gl⌢be' (page 1), which is a malformed placeholder, and Appendix D says the dataset card 'will document' licensing and maintenance, confirming that the benchmark is not currently released. Since XL-DocBench is presented as a community benchmark and its 'fully human-verified' labels must be inspectable, this blocks both use and independent verification of the paper's central contribution. The revision should provide a stable release URL with the data, annotations, and evaluation code.
  3. [§3.3] The 'fully human-verified' claim is strong, but no reliability evidence is reported. There is no inter-annotator agreement, no description of how many experts independently annotated each item, no adjudication procedure, and no information about expert selection or domain expertise. For a benchmark whose value lies in diagnostic attribution, annotation reliability is a core property. I ask for IAA statistics on a double-annotated sample (e.g., evidence-page boundaries, answer correctness, reasoning labels) and a description of the verification workflow.
minor comments (4)
  1. [Table 2 vs Abstract and §3.2] The headline numbers are inconsistent. The abstract and §3.2 report 1,103 multi-page evidence examples (72.6%), but Table 2 reports 'Multi-page human evidence. 1,103 (76.4%)'. Also, Table 2 lists max pages as 2,062 while the abstract and §3.2 cite contexts up to 2,303 pages. These should be reconciled with a clear definition of the denominator and of what counts as 'context length' for cross-document examples.
  2. [Figures 5 and 6] Figure 5 reports n=1,519 while Figure 6 reports n=1,287 with no explanation for the difference. Please state the denominator for the evidence-hit analysis, and clarify how unanswerable examples are treated in the retrieval dynamics.
  3. [Appendix H.2] In the evaluation sketch, the 'none'/'unanswerable' branch is scored by contains_abstention_phrase(pred_n). This could credit a response that includes both a concrete answer and an abstention phrase. Specify how such responses are handled, and consider requiring that the response contains no substantive answer.
  4. [Appendix E] The limitations section correctly notes 'public-document exposure in model pretraining', but the paper does not run any contamination control on the final set. This is closely related to the no-context filter validation and should be addressed, e.g., by reporting question-only accuracy of the evaluated systems on a sample of the final questions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark construction is externally grounded in human verification, and no evaluated quantity reduces to a fitted input.

full rationale

XL-DocBench is a benchmark-construction and evaluation paper, not a derivation of a modeled quantity from fitted parameters. The construction pipeline proposes candidates via LLMs and filters them, but the released labels—answers, evidence pages, reasoning types, and verification rules—are produced by 194 human experts with full document access (§3.3, Appendix I). No claim in the paper defines a predicted quantity in terms of the same data that were used to fit it. The leave-one-branch-out check (Eq. 1) is an automatic coarse filter, but the paper explicitly states that 'the released page-level evidence does not come from this coarse automatic stage. It is annotated and verified by 194 human experts with full document access' (Appendix A). There are no load-bearing self-citations: references such as PageIndex are external, and no uniqueness or ansatz is imported from prior work by the same authors. The unvalidated no-context filter (§3.1, Appendix A) is a genuine validity risk—if world-knowledge-answerable questions leak, the diagnostic attribution claim is weakened—but this is a correctness/contamination concern, not circularity, because the filter is not a fitted parameter being renamed as a prediction and the final set is externally checked by humans. The paper itself acknowledges limitations such as 'possible style bias from LLM-assisted candidate generation' (Appendix E), which is an honest limitation rather than a circular step. Accordingly, no specific circular step can be quoted, and the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper adds no physical or theoretical entities. Its assumptions are benchmark-construction choices: the no-context LLM judge, the coarse branch-level path-dependency test, the reliability of human verification without agreement metrics, and the fidelity of PDF parsing. The main hand-set parameters are evaluation tolerances and the post-verification difficulty filter.

free parameters (4)
  • OCR context cap ratio = 0.80
    All OCR evaluations truncate per-page OCR text at 80% of the model context window to reserve room for generation; this cap directly shapes the usable-context comparison between 1M, 256K, and 128K models (§4.1).
  • Difficulty-oriented final retention threshold = 1,519 of 2,104 verified candidates
    After human verification, lower-difficulty items are removed to make the benchmark harder; no objective difficulty threshold is specified, so the reported 'current systems struggle' result is partly a function of this post-verification selection (§3.3, Appendix A).
  • Numeric answer tolerance = relative_error <= 0.05
    The deterministic evaluator accepts numeric answers within 5% relative error; this affects headline accuracy for numeric and percentage answers (Appendix H.2).
  • ANLS match threshold = norm_lev_sim >= 0.8
    String answers are counted correct when normalized Levenshtein similarity is at least 0.8; this is a hand-set evaluation threshold (Appendix H.2).
assumptions (4)
  • domain assumption Leave-one-branch-out criterion at chapter/section level is a valid proxy for multi-page evidence requirement (Eq. 1).
    Used to filter candidates for path-dependency, but the paper states it does not prove the final evidence set is minimal and works only at coarse granularity (§3.1).
  • domain assumption Human expert annotation over up to 2,303-page documents is accurate without reported inter-annotator agreement.
    194 experts are said to fully verify every retained item; no reliability or agreement metrics are reported (§3.3).
  • ad hoc to paper LLM-based no-context judge reliably identifies questions answerable from world knowledge, metadata, or wording leakage.
    The largest rejection source is an LLM judge given only the question; judge accuracy is not validated (Appendix A).
  • domain assumption OCR/Markdown table/figure parsing of source PDFs preserves the evidence needed for answers.
    Parsing errors are listed as a known limitation; multimodal and table questions depend on this parse (Appendix E).

how reviews work

0 comments
Cite this review

Pith. "Pith review of XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding." pith.science (2026). https://pith.science/paper/3WGXY3OA

@misc{pith2026260800036,
  author       = {Pith},
  title        = {Pith review of: XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3WGXY3OA}},
  note         = {Machine review of arXiv:2608.00036}
}
read the original abstract

Real-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages. Some questions also require comparing related reports. Reliable long-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high -- yet most existing benchmarks still measure short-context or single-page QA. We introduce XL-DocBench, a fully human-verified benchmark for extra-long document understanding, with 1,519 retained questions from six professional domains and contexts up to 2,303 pages. XL-DocBench goes beyond page-level lookup. 1,103 examples (72.6\%) use multiple evidence pages. The final set also includes 556 questions (36.6\%) that use tables, charts, or figures, and 165 questions (10.9\%) that require evidence from multiple documents. Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format, including 218 None-answer cases. We build the benchmark with a tree-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts. By coupling extra-long professional contexts with page-level evidence and typed rules, XL-DocBench fills a gap left by prior single-page, short multi-page, or text-only long-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score. The results show that current systems still struggle with long contexts, multi-page evidence, and structured reasoning over professional documents.

Figures

Figures reproduced from arXiv: 2608.00036 by the authors.

Figure 1
Figure 1. XL-DocBench construction pipeline. Long documents and document series are parsed into hierarchical trees. The pipeline generates candidate questions from multiple chapter/section branches, refines them with concrete content, adds candidate answers and verification rules, and filters artifacts. Full verification by 194 human experts identifies the final supporting pages and quotes, checks answer support and ambiguity… view at source ↗
Figure 2
Figure 2. Reasoning labels and evidence structure in the human-verified set. (a) Distribution of the twelve reasoning types, colored by tier. (b) Mean number of expert-annotated evidence pages and related PDFs per reasoning type. 50-100 100-200 200-500 500-1000 >=1000 0 100 200 300 400 500 # questions 226 489 496 257 51 (a) Context length 1 2-5 6-10 11-20 21-50 51-100>=101 0 50 100 150 200 250 300 350 # questions 340 324 190 … view at source ↗
Figure 3
Figure 3. Composition of XL-DocBench in the final human-verified set: (a) context length per [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Diagnostic performance breakdowns for all evaluated systems. Columns separate full-image input, OCR input, and agent pipelines; rows report accuracy by context length, number of expert-annotated evidence pages, and evidence span. 26.9→37.3). The two 1M-token models non…
Figure 5
Figure 5. Figure 5: Agent answer yield versus tool-call rounds. Accuracy as the retrieval rounds increases. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Agent evidence-hit dynamics. Evidence hit rate as the retrieved/read page rounds increases. is balanced (43.03% vs. 44.09%), and Claude Opus 4.6 OCR is the only frontier one-shot reader with a small cross-document penalty (−2.53 vs. GPT-5.4 OCR’s +6.40), suggesting its…
Figure 7
Figure 7. Figure 7: Failure case study 1. The example illustrates a system failure on an evidence-grounded [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Failure case study 2. The final response is inconsistent with the expert-verified evidence. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Case study: Cross-document temporal. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Case study: Cross-document reference chain. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Case study: Comparison. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Case study: Reference chain. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Case study: Ranking. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Case study: Coverage. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Case study: Reconciliation. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Case study: Set difference. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Case study: Unanswerable. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Case study: Temporal. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Case study: Compliance. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Case study: Counterfactual. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Case study: Aggregation. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: Case study: Consistency. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

172 extracted references · 10 linked inside Pith

  1. [1]

    Pixtral 12b.arXiv preprint arXiv:2410.07073, 2024

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b.arXiv preprint arXiv:2410.07073, 2024

    arXiv 2024

  2. [2]

    Introducing claude opus 4.6

    Anthropic. Introducing claude opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6, 2026

    2026

  3. [3]

    Docformer: End-to-end transformer for document understanding

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. InICCV, pages 993–1003, 2021

    2021

  4. [4]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. InACL, pages 3119–3137, 2024

    2024

  5. [5]

    M-longdoc: A benchmark for multimodal super-long document understanding and A retrieval-aware tuning framework

    Yew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song, Chaoqun Liu, Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super-long document understanding and A retrieval-aware tuning framework. InEMNLP, pages 9233– 9250, 2025

    2025

  6. [6]

    M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding.arXiv preprint arXiv:2411.04952, 2024

    Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding.arXiv preprint arXiv:2411.04952, 2024

    arXiv 2024

  7. [7]

    Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating

    Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. InACL, pages 1135–1159, 2025

    2025

  8. [8]

    A survey on mllm-based visually rich document understanding: Methods, challenges, and emerging trends.arXiv preprint arXiv:2507.09861, 2025

    Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Geoffrey Martin, and Yifan Peng. A survey on mllm-based visually rich document understanding: Methods, challenges, and emerging trends.arXiv preprint arXiv:2507.09861, 2025

    arXiv 2025

Show all 172 references
  1. [9]

    Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449, 2024

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449, 2024

    2024 arXiv

  2. [10]

    Gemini-3.https://aistudio.google.com/models/gemini-3/, 2026

    Google. Gemini-3.https://aistudio.google.com/models/gemini-3/, 2026

    2026

  3. [11]

    Mdoca- gent: A multi-modal multi-agent framework for document understanding.arXiv preprint arXiv:2503.13964, 2025

    Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao. Mdoca- gent: A multi-modal multi-agent framework for document understanding.arXiv preprint arXiv:2503.13964, 2025

    2025 arXiv

  4. [12]

    mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. InACL, pages 5817–5834, 2025

    2025

  5. [13]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. InACM MM, pages 4083–4091, 2022. 10

    2022

  6. [14]

    Simpledoc: Multi-modal document understanding with dual-cue page retrieval and iterative refinement

    Chelsi Jain, Yiran Wu, Yifan Zeng, Jiale Liu, Shengyu Dai, Zhenwen Shao, Qingyun Wu, and Huazheng Wang. Simpledoc: Multi-modal document understanding with dual-cue page retrieval and iterative refinement. InProceedings of the 2025 Conference on Empirical Methods in Natural Lan...

    2025

  7. [15]

    Donut: Document understanding trans- former without ocr.arXiv preprint arXiv:2111.15664, 7(15):2, 2021

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding trans- former without ocr.arXiv preprint arXiv:2111.15664, 7(15):2, 2021

    2021 arXiv

  8. [16]

    Deepread: Document structure-aware reasoning to enhance agentic search.arXiv preprint arXiv:2602.05014, 2026

    Zhanli Li, Huiwen Tian, Lvzhou Luo, Yixuan Cao, and Ping Luo. Deepread: Document structure-aware reasoning to enhance agentic search.arXiv preprint arXiv:2602.05014, 2026

    2026

  9. [17]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955, 2024

    2024 arXiv

  10. [18]

    Textmon- key: An ocr-free large multimodal model for understanding document.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmon- key: An ocr-free large multimodal model for understanding document.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    2026

  11. [19]

    Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings

    Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, et al. Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings. InACL, pages 19568–19580, 2025

    2025

  12. [20]

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.NeurIPS, 37:95963–96010, 2024

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.NeurIPS, 37:95963–96010, 2024

    2024

  13. [21]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InACL, pages 2263–2279, 2022

    2022

  14. [22]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. InWACV, pages 1697–1706, 2022

    2022

  15. [23]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InCVPR, pages 2200–2209, 2021

    2021

  16. [24]

    Qwen3.5: Towards native multimodal agents

    Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id= qwen3.5, 2026

    2026

  17. [25]

    Scrolls: Standardized comparison over long language sequences

    Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. InEMNLP, pages 12007–12021, 2022

    2022

  18. [26]

    Vdocrag: Retrieval-augmented generation over visually-rich documents

    Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Vdocrag: Retrieval-augmented generation over visually-rich documents. InCVPR, pages 24827–24837, 2025

    2025

  19. [27]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In AAAI, volume 37, pages 13636–13645, 2023

    2023

  20. [28]

    Hierarchical multimodal transformers for multipage docvqa.Pattern Recognition, 144:109834, 2023

    Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage docvqa.Pattern Recognition, 144:109834, 2023

    2023

  21. [29]

    Document understanding dataset and evaluation (dude)

    Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Document understanding dataset and evaluation (dude). InICCV, pages 19528–19540, 2023

    2023

  22. [30]

    Lilt: A simple yet effective language-independent layout transformer for structured document understanding

    Jiapeng Wang, Lianwen Jin, and Kai Ding. Lilt: A simple yet effective language-independent layout transformer for structured document understanding. InACL, pages 7747–7757, 2022. 11

    2022

  23. [31]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    2025 arXiv

  24. [32]

    Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models.arXiv preprint arXiv:2406.11633, 2024

    Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, et al. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models.arXiv preprint arXiv:2406.1...

    2024

  25. [33]

    Layoutlmv2: Multi-modal pre-training for visually- rich document understanding

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually- rich document understanding. InIJCNLP, pages 2579–2591, 2021

    2021

  26. [34]

    Layoutlm: Pre-training of text and layout for document image understanding

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. InKDD, pages 1192–1200, 2020

    2020

  27. [35]

    mplug-docowl: Modularized multimodal large language model for document understanding.arXiv preprint arXiv:2307.02499, 2023

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding.arXiv preprint arXiv:2307.02499, 2023

    2023 arXiv

  28. [36]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents.arXiv preprint arXiv:2410.10594, 2024

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents.arXiv preprint arXiv:2410.10594, 2024

    2024 arXiv

  29. [37]

    Finragbench-v: A benchmark for multi- modal rag with visual citation in the financial domain

    Suifeng Zhao, Zhuoran Jin, Sujian Li, and Jun Gao. Finragbench-v: A benchmark for multi- modal rag with visual citation in the financial domain. InEMNLP, pages 4215–4249, 2025

    2025

  30. [38]

    Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. InIJNLP, pages 3277–3287, 2021

    2021

  31. [39]

    Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding and grounding

    Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat-Seng Chua. Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding and grounding. InMMM, pages 74–88, 2026

    2026

  32. [40]

    framework

    Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. Docbench: A benchmark for evaluating llm-based document reading systems. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Proce...

    2025

  33. [41]

    Context of the Model Law 1st branch

  34. [42]

    Main features of the Model Law 2nd branch

  35. [43]

    Implementation and use of the Model Law3rd branch — model stopped here ↓ descend one level — pick the 1st child

  36. [44]

    framework

    The Model Law as a "framework" law…gold answer

  37. [45]

    Banking activities

    Regulations and other laws… on p.38, parent and child sit on adjacent lines→ models read the first (parent) heading and stop, instead of descending to the child. p1 p2 p23 p24 p25 p32 p33 p38 p39 p42 p45 p51 p418 p419 Figure 7: Failure case study 1. The example illustrates a s...

  38. [47]

    Texts and abbreviations

  39. [48]

    Chronology of Jane Austen's life

  40. [49]

    The professional woman writer

  41. [50]

    Northanger Abbey, Sense and Sensibility, P […] 11 more D2 314P

  42. [52]

    A note on the second edition

  43. [53]

    Chronology of Joyce’s life

  44. [54]

    List of abbreviations

  45. [55]

    Joyce the Irishman […] 13 more D3 332P

  46. [57]

    Note on translations

  47. [58]

    Dante and the lyric past

  48. [59]

    Approaching the Vita nuova […] 17 more D4 304P

  49. [60]

    Notes on contributors

  50. [61]

    Introduction: perspectives on the African

  51. [62]

    The oral–literate interface

  52. [63]

    Chinua Achebe and the African novel

  53. [64]

    Cambridge University Press 1997

    Protest and resistance […] 13 more HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — grouped by document, manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDFS · HUMAN-CURATED, GROUPED BY DOCUMENT D1 D2 D3 D4 2 6 6 P p1 p2 p3 p4 p5 3 1 4 P p1...

    1997

  54. [65]

    Basel III: Finalising post-crisis reforms

  55. [66]

    Standardised approach for credit risk

  56. [67]

    Internal ratings-based approach for credit

  57. [68]

    Minimum capital requirements for CVA risk

  58. [69]

    Minimum capital requirements for operation

  59. [70]

    Output floor […] 1 more D2 82P

  60. [72]

    Pillar 3 disclosure requirements – consoli

  61. [73]

    Part 1: New and revised Pillar 3 disclosur

  62. [74]

    Consolidation of all existing and prospect

  63. [75]

    Two enhancements to the revised Pillar 3 f

  64. [76]

    Revisions and additions to the Pillar 3 fr […] 15 more D3 1982P

  65. [77]

    SCO Scope and definitions

  66. [78]

    CAP Definition of capital

  67. [79]

    RBC Risk-based capital requirements

  68. [80]

    CRE Calculation of RWA for credit risk

  69. [81]

    MAR Calculation of RWA for market risk

  70. [82]

    OPE Calculation of RWA for operational ris

  71. [83]

    LEV Leverage ratio […] 7 more D4 77P

  72. [84]

    Basel III: A global regulatory framework f

  73. [85]

    Strengthening the global capital framework

  74. [86]

    Introducing a global liquidity standard

  75. [87]

    Transitional arrangements

  76. [88]

    Basel III: Finalising post-crisis reforms

    Scope of application […] 10 more HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — grouped by document, manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDFS · HUMAN-CURATED, GROUPED BY DOCUMENT D1 D2 D3 D4 1 6 2 P p1 p2 p3 8 2 P p1 p2 p3 p4 ...

    2017

  77. [89]

    Financial Accounts Matrix —Transactions

  78. [90]

    Financial Accounts Matrix —Levels

  79. [91]

    Debt Growth by Sector (D.1)

  80. [92]

    Borrowing by Sector (D.2)

  81. [93]

    Debt Outstanding by Sector (D.3)

  82. [94]

    Net Wealth (B.1)

    Derivation of U.S. Net Wealth (B.1)

  83. [95]

    Distribution of Gross Domestic Product (F.2)

  84. [96]

    Distribution of National Income (F.3)

  85. [97]

    Saving and Investment (F.4)

  86. [98]

    Net Capital Transfers (F.5)

  87. [99]

    Derivation of Measures of Personal Saving (F.6)

  88. [100]

    For 2018, what is the ratio of total debt outstanding in D.3 to U.S. net wealth in B.1, expressed as a percentage?

    Assets and Liabilities of the Personal Sector (L.6) […] 104 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p21 p22 p23 p49 p99 p148 p198 S...

    2018

  89. [101]

    CHAPTER ONE INTRODUCTION

  90. [102]

    PURPOSE AND APPLICABILITY

  91. [103]

    PUBLICATION ORGANIZATION

  92. [104]

    CHAPTER TWO THE FUNDAMENTALS

  93. [105]

    CYBER RESILIENCY ENGINEERING FRAMEWORK

  94. [106]

    CYBER RESILIENCY IN THE SYSTEM LIFE CYCLE

  95. [107]

    RISK MANAGEMENT AND CYBER RESILIENCY

  96. [108]

    CHAPTER THREE CYBER RESILIENCY IN PRACTICE

  97. [109]

    SELECTING AND PRIORITIZING CYBER RESILIENCY CONSTRUCTS

  98. [110]

    ANALYTIC PRACTICES AND PROCESSES

  99. [111]

    “The purpose of this document is to supplement [SP 800-160 v1] and [SP 800-37] with guidance…

    APPENDIX A REFERENCES […] 28 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p26 p27 p28 p56 p112 p168 p224 STAGE 1 · EXPLORATION depends =...

  100. [112]

    Technical Report Documentation Page

  101. [113]

    CHAPTER 1 - INTRODUCTION

  102. [114]

    CHAPTER 2 – GROUND MOTION HAZARDS

  103. [115]

    CHAPTER 3 – GEOTECHNICAL HAZARDS

  104. [116]

    CHAPTER 4 – PRINCIPLES OF SEISMIC DESIGN

  105. [117]

    CHAPTER 5 – DEMAND ANALYSIS

  106. [120]

    Inward and Outward Spillovers and Centrality of Eur

    Box 3.5. Inward and Outward Spillovers and Centrality of Eur

  107. [121]

    Regulatory Standards for Valuations in the Insura

    Annex 3.1. Regulatory Standards for Valuations in the Insura

  108. [122]

    Data and Methodology

    Annex 3.2. Data and Methodology

  109. [123]

    insurers in Belgium, Denmark, and Spain ... Finland and Germany ... Belgium and the United Kingdom ... Netherlands ... Austria, Finland, and France

    References branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p33 p67 p101 p129 p130 p131 p135 STAGE 1 · EXPLORATION depends = [0001, 0003] "If the insurer ...

  110. [124]

    Background and Major Issues

  111. [125]

    Climate Trend Evaluation

  112. [126]

    Groundwater Monitoring Evaluation

  113. [127]

    Previous Investigations

  114. [128]

    Development of the Hydrologic System

  115. [129]

    Groundwater Availability, a Water-Budget Perspective

  116. [130]

    Climate Trends and Potential Future Effects on Groundwater A

  117. [131]

    PAGE 11

    Evaluation of Regional Groundwater-Level Monitoring Network […] 5 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p11 p12 p13 p19 p38 p57 p...

  118. [132]

    Summary Content List

  119. [133]

    Chapter 1 Summary : Narrative and life

  120. [134]

    Chapter 2 Summary : Defining narrative

  121. [135]

    Chapter 2: Defining Narrative

  122. [136]

    Chapter 3 Summary : The borders of narrative

  123. [137]

    Chapter 3: The Borders of Narrative

  124. [138]

    Chapter 4 Summary : The rhetoric of narrative

  125. [139]

    Chapter 4: The Rhetoric of Narrative

  126. [140]

    CHAPTER 6 – BEHAVIOR OF INELASTIC BRIDGES

  127. [141]

    Chapter 5 Summary : Closure

  128. [142]

    Chapter 6 Summary : Narration

  129. [143]

    Chapter 7 Summary : lnterpreting narrative

  130. [144]

    Narrative transcends art forms like novels and folk tales; it is an inherent part of everyday human communication

    Chapter 8 Summary : Three ways to interpret narrative […] 23 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p7 p8 p14 p45 p91 p137 p183 ST...

  131. [145]

    FIST Volume 2-7 Mechanical Overhaul Procedures for Hydroelec

  132. [151]

    What specific external code designation is cited in the Symbols section for any symbol definition that is also listed under Codes and Standards?

    Mechanical Database […] 40 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p7 p8 p9 p18 p36 p54 p73 STAGE 1 · EXPLORATION depends = [0002, ...

    2019

  133. [152]

    Utilisation du guide

  134. [153]

    Médicaments injectables

  135. [154]

    Solutions de perfusion

  136. [155]

    Vaccins, immunoglobulines et sérums

  137. [156]

    Médicaments à usage externe, antiseptiques et désinfectants

  138. [157]

    Médicaments potentiellement dangereux ou obsolètes ou ineffi

  139. [158]

    Principales références

  140. [159]

    In 1978, the Alma Ata conference confirmed that access to essential medicines was vital to prevent and treat diseases affecting millions of people throughout the world

    Dans la même collection branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p5 p6 p7 p131 p262 p393 p525 STAGE 1 · EXPLORATION depends = [0002, 0003] "Dans l...

    2019

  141. [160]

    DTaP/DT and Tdap/Td Vaccines Pink Book Web-on-Demand Series

  142. [162]

    Disclosure Statements

  143. [163]

    DTaP/DT and Tdap/Td Vaccines

  144. [164]

    Clinical Considerations

  145. [165]

    Storage and Handling

  146. [166]

    Continuing Education Information

  147. [167]

    E-mail Your Immunization Questions to Us

  148. [168]

    Thank You From Atlanta! branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p3 p4 p5 p39 p78 p117 p157 STAGE 1 · EXPLORATION depends = [0002, 0003] "What is ...

    2024

  149. [169]

    Final Government Distribution IPCC AR6 WGI

  150. [170]

    Effective Radiative Forcing

  151. [171]

    Two-layer energy balance model for climate emulation

  152. [172]

    Performance of emulators compared to key physical climate as

  153. [173]

    Equilibrium Climate Sensitivity and Transient Climate Respon

  154. [174]

    Tables of greenhouse gas lifetimes, radiative efficiencies a

  155. [175]

    References branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p13 p14 p15 p16 p27 p41 p55 STAGE 1 · EXPLORATION depends = [0009] "In the two-layer energy ba...

  156. [176]

    NOTICE" PAGE 3

    CHAPTER 7 – SEISMIC DESIGN METHODS […] 13 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p2 p3 p4 p152 p304 p456 p608 STAGE 1 · EXPLORATIO...

  157. [181]

    Record of Revisions Revision Number: 1

    FIST Revision Requests […] 6 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p10 p11 p16 p18 p36 p54 p73 STAGE 1 · EXPLORATION depends = [0...

  158. [182]

    Reclamation Standards and Documents

  159. [183]

    Acronyms and Abbreviations

  160. [184]

    Reclamation Standard Practices

  161. [185]

    Manufacturer Recommendations

  162. [186]

    FIST Revision Requests

  163. [187]

    Development of Maintenance Tables

  164. [188]

    Acronyms and Abbreviations

    Mechanical Maintenance of Mechanical and Digital Governors f […] 129 more sections branch A branch B HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDF · HUMAN-CURATED p13 p14 p15 p55 p111 p...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.