Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Docopilot: Improving Multimodal Models for Document-Level Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning on a new 758K-question document dataset lets an 8B multimodal model beat a 26B model on long-document QA without retrieval.

desk verdict A useful dataset-baseline paper whose headline MM-NIAH result currently needs a contamination check and a couple of benchmark-hygiene fixes before the central claim is sealed. read the letter →

arxiv 2507.14675 v1 pith:IMNA4ZR6 submitted 2025-07-19 cs.CV cs.CL

Yuchen Duan , Zhe Chen , Yusong Hu , Weiyun Wang , Shenglong Ye , Botian Shi , Lewei Lu , Qibin Hou
show 4 more authors
Tong Lu Hongsheng Li Jifeng Dai Wenhai Wang
This is my paper
classification cs.CVcs.CL
keywords multimodallargelanguagemodelsdocument-levelunderstandinglong-contextquestionansweringretrieval-freedocumentQAdatasetconstructioncross-pagereasoningfine-tuningDoc-750K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Docopilot argues that high-quality document-level training data, not a new architecture, is what multimodal large language models need to understand multi-page documents. The paper introduces Doc-750K, a dataset of 758K questions built from academic papers with real QA pairs and cross-page dependencies, and uses it to fine-tune InternVL2 into Docopilot, a retrieval-free model that reads whole documents in one pass. The headline result is that Docopilot-8B scores 61.8 on MM-NIAH, 19.9 points above its base model and above the much larger InternVL2-26B at under 31% of that model's latency. The paper also reports consistent gains on multi-page benchmarks such as MMLongBench-Doc and DocGenome while single-page VQA performance stays flat. If the evaluation transfer is fair, the work implies that long-document QA can be done without retrieval-augmented generation, with fewer failure modes and lower latency.

What carries the argument

The load-bearing object is Doc-750K, assembled by a three-stage engine: raw collection from academic sources, extraction into interleaved text-image and rendered multi-image formats, and QA-pair construction using deterministic structural tasks (abstract writing, titling, caption writing, experiment writing, translation) plus human-written peer-review reply pairs and a small model-generated remainder (4.8%). Training on 32K-token sequences is made practical by multimodal data packing, Ring Attention, and a fused kernel library, but the paper's argument is that the document-level supervision in Doc-750K, not these engineering tricks, carries the improvement.

What would settle it

Search for near-duplicate pages or papers between Doc-750K's source corpus and the documents used in MMLongBench-Doc, DocGenome, and MM-NIAH; substantial overlap would mean the headline accuracy gap could be memorization. A direct test is to run Docopilot on non-academic document types such as financial reports, legal filings, and user manuals and see whether the cross-page gains survive outside the academic domain the dataset was built on.

Watch

Extended reading notes

Core claim

The central claim is that a standard ViT-MLP-LLM model, fine-tuned on a large corpus of whole academic documents with questions that require integrating information across pages, becomes a native long-document QA system. The paper presents Doc-750K—758K questions in 251K conversations with 3.1M images—where 31.6% of QA pairs are real, structurally derived, or human-written rather than artificially inserted. On this data, Docopilot-8B reaches 61.8 on MM-NIAH, surpassing the 26B InternVL2 variant despite being an 8B model, and it improves over RAG-based baselines on MMLongBench-Doc and DocGenome while keeping inference time equal to the base model. The discovery, in the paper's telling, is that retrieval-free document-level understanding is attainable by supervision alone.

Load-bearing premise

The claimed gains assume the evaluation documents are not in the training set; the paper reports no deduplication or contamination check, so this transfer assumption is load-bearing.

Editorial extensions

If this is right

  • One forward pass over a whole document can replace multi-stage RAG for documents within the context window, removing retrieval errors and the added latency of a separate retrieval step.
  • Small models can close much of the gap to much larger ones on long-document QA when trained on the right data; Docopilot-2B approaches or exceeds InternVL2-26B on several MM-NIAH length bands.
  • The multi-page benchmark gains (for instance, +12.6% over InternVL2-8B on DocGenome multi-page QA) suggest cross-page reasoning is learnable from document-level supervision rather than requiring architectural changes.
  • Because single-page VQA scores remain roughly flat, the dataset adds long-context ability without sacrificing short-document reading performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If contamination is ruled out, the recipe likely transfers to other document genres: collecting real QA pairs from legal, medical, or financial corpora could produce similar retrieval-free gains, since the active ingredient is document-level supervision rather than academic content.
  • The proxy tasks (abstract writing, captioning, translation) may be doing more work than the raw QA pairs; ablating them separately would show whether structure-learning tasks are necessary or whether the multi-image QA data alone drives the improvement.
  • For documents longer than the 32K context window, RAG or some hierarchical compression would still be needed; the boundary between one-pass reading and retrieval is an open question this paper does not settle.
  • Removing the 4.8% model-generated QA pairs from training would test whether that small synthetic component contributes to the gains or is incidental.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Docopilot introduces Doc-750K, a 758K-question multimodal document dataset with 3.1M images collected from Arxiv, Sci-Hub, and OpenReview, presented in both interleaved text-image and rendered-page formats. The authors fine-tune InternVL2-2B and InternVL2-8B on Doc-750K mixed with open SFT datasets, using multimodal data packing, Ring Attention, and Liger Kernel, and report improvements over the base models on MP-DocVQA, MMLongBench-Doc, DocGenome, and MM-NIAH, as well as lower latency than RAG baselines. They also report single-page VQA results and an ablation showing that each Doc-750K data source contributes to MMLongBench-Doc performance.

Significance. If the reported evaluations are clean, Doc-750K is a large and potentially useful resource, and the demonstration that a retrieval-free model can gain substantially on MM-NIAH would be an important result for document-level MLLMs. The engineering contributions (data packing, Ring Attention, Liger Kernel) are clearly described, and the release of data, code, and models is a concrete strength. However, the central claim that Docopilot improves cross-page reasoning on unseen documents is currently unverified because of train-on-test overlap and the absence of overlap or deduplication analysis, so the paper needs substantive revision before its headline conclusions can be accepted.

major comments (4)
  1. [Section 3.3, Figure 4(b), Tables 4–5] The SFT recipe explicitly includes MP-DocVQA (51K), DocVQA (56K), InfoVQA (25.5K), ChartQA (30.2K), and DocReason (25.8K), yet Section 5.2 evaluates on MP-DocVQA and Section 5.4 evaluates on DocVQA, ChartQA, and InfoVQA. This makes part of Tables 4 and 5 a train-on-test evaluation, which is especially damaging for the MP-DocVQA row (81.3 in Table 4) because MP-DocVQA is among the multi-page document QA datasets used in the SFT recipe. Please remove these benchmarks from the training mixture or evaluate only on a held-out subset, and state explicitly which reported rows are unaffected by this overlap.
  2. [Sections 3.1–3.2 and 5.2–5.3] For MMLongBench-Doc, DocGenome, and MM-NIAH, no deduplication or overlap analysis against Doc-750K is reported. Doc-750K is built from Arxiv, Sci-Hub, and OpenReview academic documents, while MMLongBench-Doc and DocGenome are also academic-document benchmarks; MM-NIAH (refs [85]/[86]) has overlapping authorship with this paper. Because the headline claim—a 19.9-point MM-NIAH gain and superiority over InternVL2-26B in Table 4—depends on transfer to unseen documents, please provide quantitative overlap statistics (exact or near-duplicate documents or pages) and report the headline metrics on the deduplicated subset.
  3. [Section 3.1 and Figure 4(a)] The text states that 'only 4.8% of the data is LLM-generated,' but Figure 4(a) shows MT-QA (interleave) with 211K samples and MT-QA (image) with 211K samples, and the construction description says GPT-4o is used for other documents. If these MT-QA pairs are GPT-4o-generated, they constitute roughly 55.7% of Doc-750K's 758K questions, contradicting the 4.8% claim. If the 4.8% figure is correct, clarify what MT-QA denotes and where the hundreds of thousands of GPT-4o pairs, if any, appear in the statistics; this is needed to support the dataset-quality claims.
  4. [Sections 5.1 and Table 4] The model's maximum sequence length is set to 32K tokens (Section 5.1, token threshold 32K in Table 10), while MM-NIAH's Long category is defined as (32K, 64K] and Table 4 reports large gains there (e.g., Docopilot-8B Long 55.3 vs. InternVL2-8B 32.4). Please explain how test sequences longer than the training length are processed (truncation, positional extrapolation, or another mechanism) and whether this is a length-generalization claim or a within-training-range result; the reported numbers cannot be interpreted without this information.
minor comments (4)
  1. [References [85] and [86]] References [85] and [86] cite the same arXiv paper but are used as if they were different sources; please unify them into a single reference for MM-NIAH.
  2. [Table 5] Table 5 reports differences such as +0.4 points on DocVQA, 0 on ChartQA, and –1.5 on InfoVQA without error bars or repeated runs; the text should either add confidence intervals or soften 'outperforms' to 'performs comparably'.
  3. [Figure 1 and Table 7] The '<31% latency' claim in Figure 1 is not directly checkable from Table 7, which lists no InternVL2-26B latency; please add the InternVL2-26B latency used for the ratio.
  4. [Section 3.2] The statement that academic papers constitute approximately 32.6% of the total data is ambiguous given that Figure 4(a) reports source percentages of 55.8% Arxiv, 29.8% OpenReview, and 14.4% Sci-Hub; please clarify the denominator for the 32.6% statistic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Docopilot is an empirical dataset-and-fine-tuning paper; its reported gains are not derived from its training inputs by construction, though benchmark-overlap and co-authored-benchmark concerns are evaluation-hygiene risks.

full rationale

Docopilot contains no formal derivation chain to walk: it introduces a new dataset (Doc-750K) and fine-tunes InternVL2 on a mixed SFT recipe, then reports performance on external document benchmarks. The claimed headline result (61.8 on MM-NIAH, +19.9 over InternVL2-8B) is an empirical outcome, not a quantity computed from the training data by construction. The dataset is newly assembled from Sci-Hub, arXiv, and OpenReview, and the proxy tasks are not definitionally tied to the evaluation metrics. The Limitations section (Sec. 11) concedes the academic-document domain restriction, but that is a generalization limitation, not circularity. Two evaluation-hygiene concerns are worth noting without raising the circularity score: (1) Sec. 3.3 lists DocVQA, ChartQA, InfoVQA, and MP-DocVQA in the SFT recipe, and Sec. 5 reports results on those same benchmarks; if test splits were used for training this would be leakage, but the paper does not state the splits, so this is unverified rather than an exhibited reduction. (2) MM-NIAH (refs [85]/[86]) is co-authored by several Docopilot authors; however, the paper's central comparison is also grounded in externally published benchmarks (MMLongBench-Doc, DocGenome, MP-DocVQA, DocVQA), so the self-citation is not the sole load-bearing justification. No fitted parameter is relabeled as a prediction, no benchmark score is defined in terms of the training objective, and no uniqueness theorem is imported from the authors' prior work. I therefore find no circular step qualifying under the stated evidence rules.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claim rests on dataset quality, benchmark fairness, and evaluation fidelity rather than on mathematical derivation. The main unevidenced inputs are the quality of synthetic QA, the absence of training/eval contamination, and the reliability of GPT-4o-based scoring.

free parameters (3)
  • Max sequence length (Token Threshold Tt) = 32K tokens
    Chosen training cap; MM-NIAH Long reaches 64K, but no length extrapolation procedure is specified for inputs above 32K.
  • Image threshold Ti = 48 images
    Packing threshold chosen by hand; affects how many samples are concatenated and how truncation behaves.
  • Max tile number = 24
    Dynamic high-resolution tiling cap that affects OCR quality and visual detail for long documents.
assumptions (4)
  • domain assumption QA generation by GPT-4o yields faithful answers for the 'other documents' subset.
    Section 3.1 says GPT-4o is used for QA generation; only 500 samples were manually reviewed for relevance.
  • ad hoc to paper Evaluation benchmarks do not overlap with Doc-750K training documents.
    No deduplication or contamination analysis is reported, while training sources (arXiv, Sci-Hub, OpenReview) overlap in genre with MMLongBench-Doc and DocGenome.
  • domain assumption GPT-4o-based scoring on MMLongBench-Doc and DocGenome is consistent and unbiased.
    Section 8.1 says correctness is judged by GPT-4o; no human agreement or judge calibration is reported.
  • domain assumption Attention masking across packed samples is correct with Ring Attention and Liger Kernel.
    Section 9.2 states tokens only attend within the same original sample, but no correctness verification of the packing attention mask is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Docopilot: Improving Multimodal Models for Document-Level Understanding." pith.science (2026). https://pith.science/paper/IMNA4ZR6

@misc{pith2026250714675,
  author       = {Pith},
  title        = {Pith review of: Docopilot: Improving Multimodal Models for Document-Level Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMNA4ZR6}},
  note         = {Machine review of arXiv:2507.14675}
}
read the original abstract

Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level datasets. While current retrieval-augmented generation (RAG) methods offer partial solutions, they suffer from issues, such as fragmented retrieval contexts, multi-stage error accumulation, and extra time costs of retrieval. In this work, we present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents. This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents. Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG. Experiments demonstrate that Docopilot achieves superior coherence, accuracy, and efficiency in document understanding tasks and multi-turn interactions, setting a new baseline for document-level multimodal understanding. Data, code, and models are released at https://github.com/OpenGVLab/Docopilot

Figures

Figures reproduced from arXiv: 2507.14675 by the authors.

Figure 1
Figure 1. Accuracy v.s inference latency on MM-NIAH. The proposed Docopilot-8B shows a notable improvement over base￾line models [73], achieving a +19.9% accuracy gain compared to InternVL2-8B and surpassing InternVL2-26B with less than 31% of the inference latency. Additionally, Docopilot-2B uses fewer parameters (less than 10%) while exhibiting comparable perfor￾mance to the 10× larger InternVL2-26B. These results suggest t… view at source ↗
Figure 2
Figure 2. Multimodal document dataset generation pipeline. This pipeline involves three main stages: (1) Raw Data Collection: Documents are gathered from sources like Sci-Hub, arXiv, and OpenReview, available in PDF and HTML formats. (2) Document Content Extraction: Multimodal content is processed in two formats: interleaved text-image format and multi-image format. (3) Question-Answer Pairs Construction: QA pairs are generat… view at source ↗
Figure 3
Figure 3. Visualization of an example from Doc-750K. The left side presents the interleaved text-image format data obtained through Document Content Extraction, while the right side showcases the annotations generated via Question-Answer Pairs Construction. example, “<text>\n<image>\n<text>\n<image>” This format captures the document’s textual content, mak￾ing it easier to construct question-answer pairs. (2) Multi-Image Form… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Data distribution of our dataset. The outer circle shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Left: Data distribution of Doc-750K. Right: Data distribution of our complete SFT training dataset. Note that t…
Figure 5
Figure 5. Figure 5: An example of multi-image format document. Each page of the document is rendered as an image. Datasets #QA Image Types Tasks MP-DocVQA [76] 46K PDF Documents VQA DUDE [78] 41K PDF Documents VQA MP-DocStruct1M [27] 1M PDF Documents Text Parsing, Text Lookup MP-DocReason…
Figure 6
Figure 6. Figure 6: An example of interleaved text-image format document. We capture the documents’ textual content and construct them into interleaved images and text. Implementation Details of Multimodel RAG. Vis￾RAG [95] uses their proposed retrieval model VisRet to cal￾culate scores f…
Figure 7
Figure 7. Figure 7: The examples of different formats of the QA pairs. All tasks leverage the inherent structure of the documents [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: A qualitative example of retrieval in a multi-page document. Among the most important applications listed, "Point-of-Care Decision Support" is shown a picture of the interface as an example. Image-1: <Concat-Page 1>\n Image-2: <Concat-Page 2>\n … Image-10: <Concat-Page…
Figure 9
Figure 9. Figure 9: A qualitative example of backward query in a multi-page document [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: A qualitative example of consistency of answers in multi pages. In the "Academics and Related Resources" part of the guidebook, there are three QR codes shown. Image-1: < Concat-Page 1>\n Image-2: < Concat-Page 2>\n … Image-14: <Concat-Page 14>\n How many QR codes are…
Figure 11
Figure 11. Figure 11: A qualitative example of counting across pages [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: A qualitative example of retrieval in a specific section of a super-long document [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.

Reference graph

Works this paper leans on

104 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [86]

    Needle in a multimodal haystack

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. arXiv preprint arXiv:2406.07230, 2024. 2, 7, 8, 5

    arXiv 2024

  2. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35: 23716–23736, 2022. 2

    2022

  3. [2]

    L- eval: Instituting standardized evaluation for long context lan- guage models

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L- eval: Instituting standardized evaluation for long context lan- guage models. arXiv preprint arXiv:2307.11088, 2023. 1

    arXiv 2023

  4. [3]

    Docformerv2: Local features for document understanding

    Srikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran, Yichu Zhou, and R Manmatha. Docformerv2: Local features for document understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 709–718, 2024. 2

    2024

  5. [4]

    Wukong-reader: Multi-modal pre- training for fine-grained visual document understanding

    Haoli Bai, Zhiguang Liu, Xiaojun Meng, Wentao Li, Shuang Liu, Nian Xie, Rongfu Zheng, Liangwei Wang, Lu Hou, Jiansheng Wei, et al. Wukong-reader: Multi-modal pre- training for fine-grained visual document understanding. arXiv preprint arXiv:2212.09621, 2022. 2

    arXiv 2022

  6. [5]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  7. [6]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  8. [7]

    Longalign: A recipe for long context alignment of large language models

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058, 2024. 6

    arXiv 2024

Show all 104 references
  1. [8]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In ICCV, pages 4291–4301, 2019. 6

    2019

  2. [9]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. 1, 7

    2024 arXiv

  3. [10]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 1

    2015 arXiv

  4. [11]

    Longlora: Efficient fine- tuning of long-context large language models.arXiv preprint arXiv:2309.12307, 2023

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhi- jian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine- tuning of long-context large language models.arXiv preprint arXiv:2309.12307, 2023. 2, 3, 6

    2023 arXiv

  5. [12]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 1, 2

    2023 arXiv

  6. [13]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1, 2, 7, 8

    2024 arXiv

  7. [14]

    M3docrag: Multi-modal retrieval is what you need for multi-page multi-document understanding

    Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi-modal retrieval is what you need for multi-page multi-document understanding. arXiv preprint arXiv:2411.04952, 2024. 1, 2, 3, 7

    2024 arXiv

  8. [15]

    Simple and effec- tive multi-paragraph reading comprehension

    Christopher Clark and Matt Gardner. Simple and effec- tive multi-paragraph reading comprehension. In ACL, pages 845–855, 2018. 6

    2018

  9. [16]

    Liger kernel: Efficient triton kernels for llm training

    Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen, et al. Liger kernel: Efficient triton kernels for llm training. arXiv preprint arXiv:2410.10989, 2024. 2, 6

    2024 arXiv

  10. [17]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. 3

    2023 arXiv

  11. [18]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. NeurIPS, 35:16344–16359, 2022. 3

    2022

  12. [19]

    Longnet: Scaling transformers to 1,000,000,000 tokens

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shao- han Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. 3

    2023 arXiv

  13. [20]

    Dreamllm: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal com- prehension and creation. In ICLR, 2024. 2

    2024

  14. [21]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  15. [22]

    Colpali: Efficient doc- ument retrieval with vision language models

    Manuel Faysse, Hugues Sibille, Tony Wu, Gautier Viaud, C´eline Hudelot, and Pierre Colombo. Colpali: Efficient doc- ument retrieval with vision language models. arXiv preprint arXiv:2407.01449, 2024. 1, 3

    2024 arXiv

  16. [23]

    Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding

    Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint arXiv:2311.11810,

    arXiv

  17. [24]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1

    2023 arXiv

  18. [25]

    Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance. arXiv preprint arXiv:2410.16261, 2024. 2

    arXiv

  19. [26]

    Lm-infinite: Simple on-the-fly length generalization for large language models

    Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. 3

    2023 arXiv

  20. [27]

    mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding. arXiv preprint arXiv:2409.03420, 2024. 7, 2

    2024 arXiv

  21. [28]

    Mini-monkey: Alleviate the saw- tooth effect by multi-scale adaptive cropping

    Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alleviate the saw- tooth effect by multi-scale adaptive cropping. arXiv preprint arXiv:2408.02034, 2024. 7

    2024 arXiv

  22. [29]

    Open- clip

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip. Zenodo. Version 0.1. https://doi.org/10. 5281 / zeno...

    doi

  23. [30]

    Ef- fectiveness assessment of recent large vision-language mod- els

    Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, and Fahad Shahbaz Khan. Ef- fectiveness assessment of recent large vision-language mod- els. Visual Intelligence, 2(1):17, 2024. 1

    2024

  24. [31]

    Llmtest needleinahaystack

    Greg Kamradt. Llmtest needleinahaystack. https : / / github . com / gkamradt / LLMTest _ NeedleInAHaystack, 2024. Accessed: 2024-11-11. 1

    2024

  25. [32]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sang- doo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Confer- ence on Computer Vision , pages 498–517. Springer, 2022. 3

    2022

  26. [33]

    Building and better understanding vision- language models: insights and future directions

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 2, 6

    2024 arXiv

  27. [34]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid- dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NIPS, 36, 2024. 2

    2024

  28. [35]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 2

    2023 arXiv

  29. [36]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 7

    2024 arXiv

  30. [37]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888–12900, 2022. 2

    2022

  31. [38]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 2

    2023

  32. [39]

    Omnicorpus: An unified mul- timodal corpus of 10 billion-level images interleaved with text

    Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shen- glong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhang- wei Gao, Erfei Cui, et al. Omnicorpus: An unified mul- timodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406.08418, 2024. 2

    2024 arXiv

  33. [40]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607,

    arXiv

  34. [41]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 6

    2023 arXiv

  35. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2023. 1, 2, 6

    2023

  36. [43]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023. 2, 3, 6

    2023 arXiv

  37. [44]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 1

    2023 arXiv

  38. [45]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023. 1

    2023 arXiv

  39. [46]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 7

    2024 arXiv

  40. [47]

    Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond lan- guage

    Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond lan- guage. arXiv preprint arXiv:2305.05662, 2023. 2

    2023 arXiv

  41. [48]

    Mmdu: A multi-turn multi-image dialog under- standing benchmark and instruction-tuning dataset for lvlms

    Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog under- standing benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833, 2024. 6

    2024 arXiv

  42. [49]

    Layoutllm: Layout instruction tuning with large language models for document understanding

    Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout instruction tuning with large language models for document understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15630–15640, 2024. 3

    2024

  43. [50]

    Unifying multimodal retrieval via document screenshot embedding

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406.11251 ,

    arXiv

  44. [51]

    Mmlongbench-doc: Bench- marking long-context document understanding with visual- izations, 2024

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. Mmlongbench-doc: Bench- marking long-context document understanding with visual- izat...

    2024

  45. [52]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In ACL, pages 2263–2279, 2022. 6, 8, 1, 5

    2022

  46. [53]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021. 6, 8, 1, 5

    2021

  47. [54]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, pages 1697–1706, 2022. 6, 8, 1, 5

    2022

  48. [55]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947–952, 2019. 6

    2019

  49. [56]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn. openai.com/papers/GPTV_System_Card.pdf ,

  50. [57]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. https://openai.com/ index/gpt-4o-system-card/ , 2024. 7

    2024

  51. [58]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. 3

    2021 arXiv

  52. [59]

    Learning multiple visual domains with residual adapters

    Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. NeurIPS, 30, 2017. 2

    2017

  53. [60]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  54. [61]

    Unirag: Universal retrieval augmenta- tion for multi-modal large language models

    Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, and Jimmy Lin. Unirag: Universal retrieval augmenta- tion for multi-modal large language models. arXiv preprint arXiv:2405.10311, 2024. 1

    2024 arXiv

  55. [62]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 1

    2024 arXiv

  56. [63]

    Replug: Retrieval-augmented black-box language mod- els

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language mod- els. arXiv preprint arXiv:2301.12652, 2023. 1

    2023 arXiv

  57. [64]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In ECCV, pages 742–758,

  58. [65]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 1, 6

    2019

  59. [66]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  60. [67]

    Generative pretraining in multi- modality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. In ICLR, 2024. 2

    2024

  61. [68]

    A length-extrapolatable transformer

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shao- han Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554, 2022. 3

    2022 arXiv

  62. [69]

    Mohammad Reza Taesiri. Arxivqa. https://github. com/taesiri/ArXivQA, 2024. 6

    2024

  63. [70]

    Vi- sualmrc: Machine reading comprehension on document im- ages

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Vi- sualmrc: Machine reading comprehension on document im- ages. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13878–13888, 2021. 6

    2021

  64. [71]

    Unifying vision, text, and layout for universal document processing

    Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mo- hit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , ...

    2023

  65. [72]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  66. [73]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy,

    OpenGVLab Team. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy,

  67. [74]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 1

    2024

  68. [75]

    Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer

    Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401.10208, 2024. 2

    2024 arXiv

  69. [76]

    Hi- erarchical multimodal transformers for multipage docvqa

    Rub `en Tito, Dimosthenis Karatzas, and Ernest Valveny. Hi- erarchical multimodal transformers for multipage docvqa. Pattern Recognition, 144:109834, 2023. 1, 2, 7, 5

    2023

  70. [77]

    An overview of large ai models and their applications

    Xiaoguang Tu, Zhi He, Yi Huang, Zhi-Hao Zhang, Ming Yang, and Jian Zhao. An overview of large ai models and their applications. Visual Intelligence, 2(1):1–22, 2024. 1

    2024

  71. [78]

    Document understanding dataset and evalua- tion (dude)

    Jordy Van Landeghem, Ruben Tito, Łukasz Borchmann, Michał Pietruszka, Paweł Joziak, Rafał Powalski, Dawid Ju- rkiewicz, Mickael Coustaty, Bertrand Ackaert, Ernest Val- veny, et al. Document understanding dataset and evalua- tion (dude). In Proceedings IEEE/CVF international co...

    2023

  72. [79]

    Mineru: An open-source solution for precise document content extraction

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839, 2024. 3

    2024 arXiv

  73. [80]

    Docllm: A layout-aware genera- tive language model for multimodal document understand- ing

    Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nour- bakhsh, and Xiaomo Liu. Docllm: A layout-aware genera- tive language model for multimodal document understand- ing. arXiv preprint arXiv:2401.00908, 2023. 2

    2023 arXiv

  74. [81]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1

    2024 arXiv

  75. [82]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 1

    2023 arXiv

  76. [83]

    The all-seeing project v2: Towards general relation comprehension of the open world

    Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. arXiv preprint arXiv:2402.19474, 2024. 1, 2

    2024 arXiv

  77. [84]

    The all-seeing project: Towards panoptic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhen- hang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. In ICLR, 2024. 1, 2

    2024

  78. [87]

    Towards improving document understanding: An exploration on text-grounding via mllms

    Yonghui Wang, Wengang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li. Towards improving document understanding: An exploration on text-grounding via mllms. arXiv preprint arXiv:2311.13194, 2023. 3

    2023 arXiv

  79. [88]

    Vary: Scaling up the vision vocabulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109,

    arXiv

  80. [89]

    Docgenome: An open large- scale scientific document benchmark for training and test- ing multi-modal large language models

    Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wen- jie Wu, Hancheng Ye, et al. Docgenome: An open large- scale scientific document benchmark for training and test- ing multi-modal large language models. arXiv preprint arXiv:...

    2024

  81. [90]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. 3

    2023 arXiv

  82. [91]

    Longqlora: Efficient and effective method to extend context length of large language models

    Jianxin Yang. Longqlora: Efficient and effective method to extend context length of large language models. arXiv preprint arXiv:2311.04879, 2023. 2, 6

    2023 arXiv

  83. [92]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 1, 7, 8

    2024 arXiv

  84. [93]

    mplug-docowl: Modularized multimodal large language model for document understanding

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Jun- feng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023. 6

    2023 arXiv

  85. [94]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023. 3

    arXiv

  86. [95]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024. 1, 2, 3, 7, 8

    arXiv

  87. [96]

    Longcite: Enabling llms to generate fine-grained cita- tions in long-context qa

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. Longcite: Enabling llms to generate fine-grained cita- tions in long-context qa. arXiv e-prints, pages arXiv–2409,

  88. [97]

    Longreward: Improving long-context large language models with ai feedback

    Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longreward: Improving long-context large language models with ai feedback. arXiv preprint arXiv:2410.21252, 2024. 2, 6

    arXiv

  89. [98]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 7

    2024 arXiv

  90. [99]

    Multimodal c4: An open, billion-scale corpus of images interleaved with text

    Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. NIPS, 36, 2024. 2 Docopilot: Improving Multimodal...

    2024

  91. [100]

    According to the original text

    Details of Data Construction 7.1. Document Data Format As stated in Section 3.1, documents in Doc-750K can be extracted in two formats: Interleaved Text-Image Format and rendered Multi-Image Format. The interleaved for- mat utilizes an external PDF parser to extract text direc...

  92. [101]

    type": "text

    Evaluation Details 8.1. Benchmark Metrics We report the metrics of benchmarks used in the evaluation in Table 9. For DocVQA [53], InfoVQA [54], and MP- DocVQA [76], we employ ANLS to evaluate the similarity between model responses and ground truth answers, while ChartQA [52] u...

  93. [102]

    Experiments

    Training Details 9.1. Hyperparameters We report the models and training hyperparameters of Docopilot-2B and Docopilot-8B in Table 10. 9.2. Multimodal Packed Dataset In this section, we provide a detailed description of our packing algorithm. The main workflow is outlined in Al...

  94. [103]

    Each figure highlights a specific capability of the model in addressing various tasks

    Qualitative Examples In this section, we show a series of qualitative examples to illustrate the effectiveness of our Docopilot in handling complex multi-page documents. Each figure highlights a specific capability of the model in addressing various tasks. Figure 8 demonstrate...

  95. [104]

    3","3","3

    Limitations The objective of Doc-750K is to develop a large-scale, multi-task, multimodal document-level QA dataset that efficiently trains MLLMs for document understanding. Sourced primarily from open academic platforms, the dataset focuses on tasks like multi-page QA, reason...

  96. [105]

    Who are you talking to

    **Observe your clients**: … Image-1: <Concat-Page 1>\n Image-2: < Concat-Page 2>\n … Image-17: < Concat-Page 17>\n In the "Who are you talking to" section, what tips does the author give us? Line them up,Observe your clients,Build a buyer persona. ['line them up', 'observe you...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.