Pith. sign in

REVIEW 5 major objections 7 minor 2 cited by

GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling

T0 review · 5 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-axis benchmark isolates whether document AI fails at seeing or at reasoning, and a 1%-parameter fix lifts weak areas without forgetting others.

desk verdict A useful benchmark idea and a cheap fine-tuning trick, but the decoupling isn't validated and the cross-domain results contradict the paper's own claims. read the letter →

arxiv 2505.00063 v2 pith:G2OND4CF submitted 2025-04-30 cs.CL cs.CV

Siqi Li , Yufan Shen , Xiangnan Chen , Jiayi Chen , Hengwei Ju , Haodong Duan , Song Mao , Hongbin Zhou
show 8 more authors
Bo Zhang Bin Fu Pinlong Cai Licheng Wen Botian Shi Yong Liu Xinyu Cai Yu Qiao
This is my paper · ORCID
classification cs.CLcs.CV
keywords documentintelligencebenchmarkmultimodallargelanguagemodelscomplexitydecouplingvisualreasoningcatastrophicforgettingparameterfreezing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces GDI-Bench, a benchmark that labels document tasks by visual complexity (V0–V2) and reasoning complexity (R0–R2), forming a 3x3 grid over 2.3k images and 19 tasks. The authors argue that this decoupling lets evaluators pinpoint whether a model's failure comes from perception or from higher-level reasoning, which ordinary single-score benchmarks cannot do. They also propose LW-AFT, a training method that freezes about 99% of a model's parameters and updates only a small domain-sensitive subset, reducing catastrophic forgetting during supervised fine-tuning. Using this method on the InternVL3-8B model, they produce a GDI-Model that keeps the base model's strong R0 performance while improving its R1 and R2 reasoning scores. If the claims hold, the benchmark offers a practical diagnostic grid for document intelligence and a cheap recipe for targeted model improvement.

What carries the argument

The central object is the 3x3 difficulty grid of GDI-Bench, where each test case is labeled by visual complexity (V0/V1/V2) and reasoning complexity (R0/R1/R2). The visual labels are derived from a data-driven rule: domains whose OmniDocBench end-to-end edit-distance scores exceed 0.142 are classed as V2, and the rest as V1. The reasoning labels come from task design, with R0 being verbatim page extraction, R1 requiring selective information retrieval, and R2 requiring logical or multi-element inference. The training method LW-AFT is the carrying mechanism for the repair claim: it first fine-tunes a small expert model on a mini-set to measure per-parameter absolute changes, then computes per-layer average change magnitudes, allocates a global unfrozen-parameter budget H across layers proportionally to those magnitudes, and finally masks gradients so that only the top h_l parameters in each layer update. This parameter-freezing mask is what preserves R0 skills while allowing R1 and R2 improvements, directly connecting the benchmark's diagnostic output to a concrete optimization strategy.

What would settle it

Recompute the V0/V1/V2 labels for the nine document categories using a substantially improved OCR/parsing pipeline and check whether the categories assigned as V2 change: if, say, textbook and exam-paper domains no longer have edit distances above the 0.142 threshold, the visual axis is tracking current model difficulty rather than stable visual complexity, undermining the claimed decoupling. A second check would be to verify with human annotators that V2 documents are genuinely harder to parse visually than V1 documents after controlling for content length and language.

Watch

Extended reading notes

Core claim

The paper's central claim is that decoupling document understanding into a visual-complexity axis and a reasoning-complexity axis exposes weaknesses that single-score benchmarks hide. Concretely, GDI-Bench assigns each task a V level (V0: plain text, V1: formal representations like tables and equations, V2: explanatory representations such as charts and complex layouts) and an R level (R0: full-page structured extraction, R1: information extraction, R2: reasoning). Evaluation on this grid shows, for example, that InternVL3-8B is strong at R0 but degrades sharply at R1 and R2. The paper further claims that this weakness localization is actionable: by analyzing parameter changes during full-parameter fine-tuning, they find over 95% of parameters barely move while a sparse 5% subset changes significantly, and they leverage this to freeze 99% of the model and update only the top 1% of sensitive parameters per layer (LW-AFT). The resulting GDI-Model maintains the base model's R0 accuracy, improves R1 and R2, and outperforms much larger models such as Qwen2.5-VL-72B at higher reasoning levels. Thus the paper simultaneously offers a diagnostic benchmark and a training method that turns diagnosis into targeted repair.

Load-bearing premise

The benchmark's visual-complexity labels are assigned from current model and pipeline performance (edit-distance above 0.142 on OmniDocBench means V2), not from any intrinsic measure of document structure, so the V axis may already encode the strengths and weaknesses of the very models being evaluated.

Editorial extensions

If this is right

  • GDI-Bench can serve as a diagnostic that maps a document model's failure mode onto one of six coordinates (V-level times R-level), guiding developers to either improve visual encoders or strengthen reasoning layers.
  • LW-AFT shows that updating only about 1% of parameters suffices to repair reasoning-level weaknesses without destroying existing extraction skills, offering a data-efficient and compute-light alternative to full fine-tuning.
  • The GDI-Model, at 8B parameters, matches or exceeds the reasoning performance of the 72B Qwen2.5-VL model on GDI-Bench, suggesting that targeted adaptation of a smaller base model can rival much larger general-purpose models on document-specific tasks.
  • The benchmark's task filtering pipeline — removing questions solvable without the image via DeepSeek-R1 and human review — gives a template for constructing vision-grounded QA data that tests genuine multimodal understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to build analogous V×R grids for other multimodal domains, such as medical imaging, UI screenshots, or video frames, where perception and reasoning failures are also confounded; the same decoupling logic should expose weakness patterns there.
  • The visual-complexity labels depend on current OCR pipeline performance, so the benchmark's V axis may drift as OCR improves; one testable consequence is that a substantially better OCR engine would re-classify some V2 domains as V1, which would change the reported weakness landscape.
  • The sparse-update observation that '95% of parameters barely move' suggests a broader hypothesis: for many SFT tasks, only a small task-salient subnetwork needs to be adjusted; if true, sensitivity-based masking could replace heavier continual-learning methods across a range of fine-tuning scenarios.
  • Because the training data for R1/R2 is explicitly sourced from domains disjoint from GDI-Bench, the benchmark could be used to measure how well a fine-tuned model transfers to unseen document types — a property the current experiments only partially probe.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces GDI-Bench, a document intelligence benchmark containing 3,660 test cases over 2.3k images across 9 scenarios and 19 tasks, with a claimed decoupling of visual complexity (V0-V2) and reasoning complexity (R0-R2). It also proposes LW-AFT (Layer-wise Adaptive Freeze-Tuning), which identifies a sparse subset of parameters for updating during SFT, and a GDI-Model fine-tuned from InternVL3-8B. The experiments report that GDI-Model improves on the base model and on LoRA fine-tuning for GDI-Bench while roughly preserving performance on existing benchmarks (Table 3), and the authors claim that GDI-Bench's difficulty decoupling enables weakness localization. The paper's central claims are that the benchmark's two difficulty axes are valid and independent, that the R1/R2 weakness of the base model is precisely identified, and that updating about 1% of parameters suffices to fix it without catastrophic forgetting. The submitted manuscript does not fully establish the validity of the difficulty axes and contains internal inconsistencies in the cross-domain/cross-task experiments.

Significance. If the main claims hold, GDI-Bench would be a useful diagnostic tool for document MLLMs, and LW-AFT would offer a simple, parameter-efficient way to improve targeted weaknesses while avoiding forgetting. The paper has several strengths: it covers a broad set of realistic document tasks, includes a human-verification pipeline for the annotations, evaluates multiple open and closed models, and reports results on standard benchmarks showing that LW-AFT largely preserves the base model's accuracy. The proposed method is simple and reproducible in principle, and the authors state an intention to open-source the benchmark and models. However, the validity of the V/R decoupling is not established, the R-axis is non-monotonic in the reported results, the V-axis is built partly from model-performance thresholds, and the cross-domain/cross-task experiments contradict the accompanying text. These issues are load-bearing because the paper's contribution is precisely the diagnostic and localization value of the benchmark and the claimed effectiveness of LW-AFT for cross-domain generalization.

major comments (5)
  1. [§3.1.1, §3.2.2, Fig. 3] The visual-complexity taxonomy is inconsistent with the stated threshold and is partially circular. Section 3.2.2 says that domains with end-to-end edit-distance scores above 0.142 are categorized as V2, but Fig. 3 reports textbook at 0.102, below the threshold, while §3.1.1 explicitly designates textbook as V2. Furthermore, the 0.142 threshold is derived from the performance of current SOTA models and pipeline tools on OmniDocBench, and those same model families are then evaluated on GDI-Bench; this means the V-axis may re-encode the performance of the models the benchmark is meant to diagnose. Please replace the model-performance threshold with structural, content-based, or human-annotated visual-complexity criteria (or provide a convincing argument that model performance is a stable proxy for intrinsic visual complexity), and resolve the textbook/threshold inconsistency.
  2. [§3.1.2, Table 5] The reasoning-complexity ordering is not supported by the reported data. In Table 5, every evaluated model scores higher on R2V0 than on R1V0 (e.g., InternVL3-8B: R2V0=0.89 vs. R1V0=0.35; Qwen2.5-VL-72B: 0.90 vs. 0.60). Because R1 is scored with 1−NED on free-form extraction while R2 is mostly single-choice exact match, the apparent R1<R2 ordering may be a metric artifact rather than a true ordering of reasoning difficulty. Without an independent validation of the R-axis (for example, human difficulty ratings or a task design that equalizes the output format across R1 and R2), the central claim that R0, R1, and R2 measure increasing reasoning complexity is not established, and the conclusion that the base model has a specific reasoning weakness in R1 is not grounded.
  3. [§4, Eq. (3)] The 'theoretical proposition' in Eq. (3) is stated as a formal claim but is not proven; it is essentially an adaptation of the Lottery Ticket Hypothesis to MLLMs with an unspecified domain transformation ψ. As written, the paper asks the reader to accept the existence of a sparse task-salient subnetwork and a transformation ψ that makes the subnetwork comparable to the full model, but ψ is never defined, instantiated, or empirically tested. Please re-label this as a motivating hypothesis and provide supporting evidence (e.g., verify for multiple models and tasks that updating only the selected top-parameter subset matches full-model updates), or remove the theorem-like presentation.
  4. [§5.1.3, Table 4] The results in Table 4 contradict the accompanying text. The text states that LW-AFT 'demonstrates strong cross-domain and cross-task capabilities, significantly outperforming the LoRA fine-tuning,' but Table 4 shows LoRA outperforming LW-AFT on all four transfer tasks (e.g., T2: 0.473 vs. 0.365; T4 date: 0.093 vs. 0.010), and LW-AFT is also worse than the base model on T2 and T3. This discrepancy directly undermines the generalization claim for LW-AFT. Please report the correct interpretation of Table 4, and either supply additional evidence for cross-domain/cross-task transfer or revise the conclusion to reflect the actual results.
  5. [§3.2.2, §5.2] The R1/R2 question-answer pairs are generated by GPT-4o, and GPT-4o is then evaluated on the same benchmark (Table 5). The authors filter out cases answerable without the image using DeepSeek-R1 and state that PhD-level annotators review the data, but the paper does not report the number of annotators, the fraction of cases revised, or any inter-annotator agreement. If the human review did not independently verify all ground-truth answers, the benchmark may inherit GPT-4o's annotation errors, which would inflate or distort model comparisons. Please add annotation quality statistics and a contamination analysis for the models that were used in the annotation loop.
minor comments (7)
  1. [Fig. 2] The legend includes a 'random' baseline, but the text never explains how this baseline is computed. Please clarify what the random score is (e.g., 25% chance for four-option questions) and how it is applied to the R0 and R1 non-choice tasks.
  2. [Fig. 3] The caption calls these 'visual complexity scores,' but the values are actually end-to-end edit distances from OmniDocBench. Please label the axis accordingly and mark the 0.142 threshold used in Section 3.2.2 so the reader can see which domains fall above and below it.
  3. [§5.1.1, Fig. 9] The freeze-rate ablation reports GDI-Bench score and several other benchmarks, but no confidence intervals or significance tests are provided. The differences between adjoining freeze rates are small, so please add error bars or statistical tests to support the claim that 99% freezing is optimal.
  4. [§3.2.2] Please specify the number of annotators and the inter-annotator agreement for the human-correct step. This is important for establishing the reliability of the benchmark's ground truth, especially since part of it is generated by GPT-4o.
  5. [§4, Eq. (3)] The metric P(·) in Eq. (3) is never defined, and the domain transformation ψ is not used again anywhere in the paper. Please define both symbols or remove them if they are not operational.
  6. [Eqs. (5)-(6)] Eq. (5) allocates h_ℓ unfrozen parameters per layer, but the text does not state how h_ℓ is rounded to an integer or how ties are handled in the TopK selection. Please clarify these implementation details.
  7. [Throughout] There are several typos and stylistic issues, including the run-on sentence in Section 4 ("the model performs well ... To address the model's weaknesses ... constructed supervised fine-tuning"), the inconsistent use of "GDIBench" vs. "GDI-Bench", and the hyphenation of "intelligence-preserving." I recommend a careful proofreading pass.

Circularity Check

1 steps flagged · score 5.0 of 10

Visual-difficulty axis is built from SOTA model performance, so V-level results are partly circular.

  1. self definitional [Section 3.2.2 (Data Construction); also Section 3.1.1 and Fig. 3]
    "Based on end-to-end edit distance scores from SOTA models and pipeline tools on OmniDocBench, domains with scores above 0.142 are categorized as V2, indicating high visual complexity. The remaining samples are assigned to V1."

    V2 is defined by thresholding the edit-distance performance of SOTA models and pipeline tools on OmniDocBench, so the visual-difficulty axis is a relabeling of model performance rather than an independent structural measure. The later finding that models score lower on V2 than on V1 is therefore built into the label definition for the calibrating models, and the claimed decoupling of visual from reasoning complexity is not independently established by GDI-Bench. The R axis and the LW-AFT experiments are not affected by this particular circularity.

full rationale

The only load-bearing circular step is the construction of the V2 visual-complexity label. Section 3.2.2 assigns V2 to OmniDocBench domains whose end-to-end edit-distance scores from current SOTA models and pipeline tools exceed 0.142, and Fig. 3 presents these same edit-distance numbers as 'visual complexity scores.' Thus the V axis encodes the performance of the model families GDI-Bench is meant to diagnose; any V2-versus-V1 gap in Fig. 2 or Table 5 is at least partly guaranteed by the threshold rather than independently discovered. There is also an internal inconsistency: the text designates textbook as V2, although Fig. 3 reports its score (0.102) below the 0.142 threshold. I did not count the R-axis non-monotonicity (e.g., R2V0 > R1V0 for every model in Table 5) as circularity: that is a validity threat about metric comparability (1-NED for R1 vs. multiple-choice accuracy for R2) rather than a reduction of an output to an input. The LW-AFT results are supported by external benchmarks (DocVQA, ChartQA, AI2D, TextVQA, etc.) and by controlled comparisons, so the training-method contribution is not circular. Overall, the benchmark's V-dimension premise is partially self-definitional, but the R dimension, the dataset itself, and the LW-AFT experiments carry independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claims rest on four free parameters (V2 threshold, alpha, freeze rate, sparsity cutoff), an unproven sparse-subnetwork assertion that motivates LW-AFT, model-dependent definitions of visual complexity and ground truth, and an unquantified separation between training and test document sources.

free parameters (4)
  • V2 visual-complexity threshold = 0.142 edit distance
    Section 3.2.2: domains with OmniDocBench end-to-end edit-distance scores above 0.142 are classified as V2; this hand-picked cut controls the benchmark's difficulty taxonomy.
  • Expert training subset alpha = 10% of the full dataset
    Section 5.1.1 and Fig. 8: alpha is selected by ablation to maximize GDI-Bench score; it controls the mini-set used to estimate parameter sensitivity.
  • Freeze rate = 99% frozen, H = 1% of parameters
    Section 5.1.1 and Fig. 9: the freeze rate is selected by ablation as the best trade-off between GDI-Bench performance and generalization.
  • Sparse-subset significance cutoff = 0.005 parameter difference
    Fig. 5 uses this cutoff to state that a 'critical sparse subset (5%)' undergoes substantial modification; the cutoff itself is arbitrary and used for analysis rather than for the final mask selection.
assumptions (4)
  • ad hoc to paper For any pretrained MLLM there exists a sparse task-salient subnetwork and a domain transformation psi such that updating the subnetwork matches the full model on the target domain.
    Eq. 3 is introduced as a 'theoretical proposition' but no proof is given; it is an imported Lottery-Ticket-style assumption that justifies why LW-AFT may freeze most parameters.
  • domain assumption Performance gaps of current SOTA models on OmniDocBench are a valid measure of intrinsic visual complexity.
    Section 3.1.1 uses OmniDocBench edit-distance scores to assign V1/V2 labels, assuming that model difficulty tracks document complexity rather than model quirks or annotation artifacts.
  • domain assumption GPT-4o-generated extractive and reasoning QA pairs, after automatic filtering and PhD review, are valid ground truth for R1/R2 tasks.
    Section 3.2.2 relies on AI generation plus human review; no inter-annotator agreement, error analysis, or quality metrics for the generated questions are reported.
  • domain assumption The SFT training set does not leak into GDI-Bench despite overlapping document categories.
    Appendix A.2 claims 'data sources are strictly different' from GDI-Bench, but GDI-Bench also includes newspaper, exam paper, financial report, and scientific paper images; category and task overlap is not quantified.
invented entities (1)
  • Sparse task-salient subnetwork with domain transformation psi
    purpose: To justify restricting gradient updates to a small parameter subset in LW-AFT
    Introduced in Eq. 3 as a 'theoretical proposition'; no proof, no independent measurement, and no falsifiable handle outside the paper is provided for the subnetwork or the transformation psi.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling." pith.science (2026). https://pith.science/paper/G2OND4CF

@misc{pith2026250500063,
  author       = {Pith},
  title        = {Pith review of: GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2OND4CF}},
  note         = {Machine review of arXiv:2505.00063}
}
read the original abstract

The rapid advancement of multimodal large language models (MLLMs) has profoundly impacted the document domain, creating a wide array of application scenarios. This progress highlights the need for a comprehensive benchmark to evaluate these models' capabilities across various document-specific tasks. However, existing benchmarks often fail to locate specific model weaknesses or guide systematic improvements. To bridge this gap, we introduce a General Document Intelligence Benchmark (GDI-Bench), featuring 2.3k images across 9 key scenarios and 19 document-specific tasks. By decoupling visual complexity and reasoning complexity, the GDI-Bench structures graded tasks that allow performance assessment by difficulty, aiding in model weakness identification and optimization guidance. We evaluate various open-source and closed-source models on GDI-Bench, conducting decoupled analyses in the visual and reasoning domains, revealing their strengths and weaknesses. To address the diverse tasks and domains in the GDI-Bench, we propose a GDI-Model that mitigates catastrophic forgetting during the supervised fine-tuning (SFT) process through an intelligence-preserving training strategy, thereby reinforcing the inherent weaknesses of the base model. Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench. Both our benchmark and models are or will be open-sourced on https://huggingface.co/GDIBench.

Figures

Figures reproduced from arXiv: 2505.00063 by the authors.

Figure 1
Figure 1. Overview of GDI-Bench. The benchmark decouples document understanding complexity [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of various open-source and closed-source models on GDI-Bench at different [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of visual complexity scores [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Annotation Process of GDI-Bench. For task construction, R0 tasks are gen￾erated using the original annotations or by synthesizing Markdown representa￾tions through MinerU [20]. To create R1 and R2 tasks, both the Markdown and corresponding images are input into GPT-4o …
Figure 5
Figure 5. Figure 5: Distribution of parameter dif￾ferences before and after full-parameter SFT. VIT MLP LLM 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 mean values of parameter changes [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Overview of the Layer-wise Adaptive Freeze-Tuning method. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Ablation study on the α. 0.78 0.76 0.74 0.72 0.70 0.68 GDI-Bench Score InternVL3 -8B 99.5 99 98 97 95 Freeze Rate (%) 0.70 0.75 0.80 0.85 0.90 0.95 OCR Basic Ability Score GDI-Bench DocVQA-val ChartQA-test AI2D-test TextVQA-val InfoVQA-val [PITH_FULL_IMAGE:figures/ful…
Figure 10
Figure 10. Figure 10: Table extraction task, both the InternVL3-8B model and the Qwen2.5-VL-72B model [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Newspaper titles extraction task, both full parameter fine-tuned models and LoRA fine [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Task of extracting and organizing test points into JSON format, the LoRA model fails to [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: The task of extracting and organizing problems by question number into JSON format, the [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: The task of extracting paper author information and organizing it into JSON format, [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Table reasoning tasks in scientific paper page, the GPT-4o, Grok-2-Vision-on, and [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: The task of converting chart data within a region into a markdown table, the GPT-4o model [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A survey organizes multimodal reasoning research into a staged roadmap and proposes native large multimodal reasoning models that unify perception, generation, and agentic planning.

  2. Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models

    cs.CL 2025-05 conditional novelty 2.0 of 10

    A survey-style position paper claims that reinforcement fine-tuning powers reasoning in multimodal LLMs, summarizing over a hundred recent works and proposing five future research directions.

Reference graph

Works this paper leans on

59 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  5. [5]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  6. [6]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  7. [7]

    Autohallusion: Automatic generation of hallucination benchmarks for vision-language models, 2024

    Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Lee Boyd-Graber, Tianyi Zhou, and Dinesh Manocha. Autohallusion: Automatic generation of hallucination benchmarks for vision-language models, 2024

    work page 2024

  8. [8]

    Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    work page 2024

Show all 59 references
  1. [9]

    Seed-bench- 2: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench- 2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023

    2023 arXiv

  2. [10]

    Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790, 2024

    2024 arXiv

  3. [11]

    Tabpedia: Towards comprehensive visual table understanding with concept synergy

    Weichao Zhao, Hao Feng, Qi Liu, Jingqun Tang, Binghong Wu, Lei Liao, Shu Wei, Yongjie Ye, Hao Liu, Wengang Zhou, et al. Tabpedia: Towards comprehensive visual table understanding with concept synergy. Advances in Neural Information Processing Systems, 37:7185–7212, 2024

    2024

  4. [12]

    Docpe- dia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding

    Hao Feng, Qi Liu, Hao Liu, Jingqun Tang, Wengang Zhou, Houqiang Li, and Can Huang. Docpe- dia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. Science China Information Sciences, pages 1–14, 2024

    2024

  5. [13]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  6. [14]

    DuReadervis: A Chinese dataset for open-domain document visual question answering

    Le Qi, Shangwen Lv, Hongyu Li, Jing Liu, Yu Zhang, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ting Liu. DuReadervis: A Chinese dataset for open-domain document visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1338–1351, 2022

    2022

  7. [15]

    Visualmrc: Machine reading comprehension on document images

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmrc: Machine reading comprehension on document images. In AAAI, 2021

    2021

  8. [16]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022

    2022

  9. [17]

    Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning, 2024

    Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, Jingqun Tang, Wei Chen, Lianwen Jin, Yuliang Liu, and Xiang Bai. ...

    2024

  10. [19]

    Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations, 2024

    Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. Omnidocbench: Benchmarking diverse pdf do...

    2024

  11. [20]

    Mineru: An open-source solution for precise document content extraction

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839, 2024

    2024 arXiv

  12. [21]

    Nougat: Neural optical under- standing for academic documents

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. Nougat: Neural optical under- standing for academic documents. arXiv preprint arXiv:2308.13418, 2023

    2023 arXiv

  13. [22]

    Pp-ocrv2: Bag of tricks for ultra lightweight ocr system

    Yuning Du, Chenxia Li, Ruoyu Guo, Cheng Cui, Weiwei Liu, Jun Zhou, Bin Lu, Yehua Yang, Qiwen Liu, Xiaoguang Hu, et al. Pp-ocrv2: Bag of tricks for ultra lightweight ocr system. arXiv preprint arXiv:2109.03144, 2021

    2021 arXiv

  14. [23]

    Publaynet: largest dataset ever for document layout analysis

    Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022. IEEE, 2019

    2019

  15. [24]

    Detecting text in natural image with connectionist text proposal network

    Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao. Detecting text in natural image with connectionist text proposal network. In European conference on computer vision, pages 56–72. Springer, 2016

    2016

  16. [25]

    Textboxes: A fast text detector with a single deep neural network

    Minghui Liao, Baoguang Shi, Xiang Bai, Cong Wang, Tong Lu, and Tao Mei. Textboxes: A fast text detector with a single deep neural network. InProceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017

    2017

  17. [26]

    East: An efficient and accurate scene text detector

    Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: An efficient and accurate scene text detector. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  18. [27]

    Curved scene text detection via transverse and longitudinal sequence connection

    Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition, 90:337–345, 2019

    2019

  19. [28]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  20. [29]

    Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks

    Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In International Conference on Machine Learning (ICML), 2006

    2006

  21. [30]

    Trocr: Transformer-based optical character recognition with pre-trained models

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13...

    2023

  22. [31]

    General ocr theory: Towards ocr-2.0 via a unified end-to-end model, 2024

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model, 2024. URL https://arxiv. org/abs/2409, 1704

    2024

  23. [32]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  24. [33]

    Vary: Scaling up the vision vocabulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109, 2023

    2023 arXiv

  25. [34]

    mplug-docowl: Modularized multimodal large language model for document understanding

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023

    2023 arXiv

  26. [35]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  27. [36]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024

    2024 arXiv

  28. [37]

    Focus anywhere for fine-grained multi-page document understanding

    Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Focus anywhere for fine-grained multi-page document understanding. arXiv preprint arXiv:2405.14295, 2024

    2024 arXiv

  29. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  30. [39]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023

    arXiv

  31. [40]

    Lamol: Language modeling for lifelong language learning

    Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. Lamol: Language modeling for lifelong language learning. In Proceedings of the International Conference on Learning Representations, ICLR, 2020

    2020

  32. [41]

    Rational LAMOL: A rationale-based lifelong learning framework

    Kasidis Kanwatchara, Thanapapas Horsuwan, Piyawat Lertvittayakumjorn, Boonserm Kijsirikul, and Peerapon Vateekul. Rational LAMOL: A rationale-based lifelong learning framework. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1...

    2021

  33. [42]

    Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Associati...

    1932

  34. [43]

    Progressive prompts: Continual learning for language models

    Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301.12314, 2023

    2023 arXiv

  35. [44]

    Teamwork is not always good: An empirical study of classifier drift in class-incremental information extraction

    Minqian Liu and Lifu Huang. Teamwork is not always good: An empirical study of classifier drift in class-incremental information extraction. arXiv preprint arXiv:2305.16559, 2023

    2023 arXiv

  36. [45]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  37. [46]

    Continual sequence generation with adaptive compositional modules

    Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. Continual sequence generation with adaptive compositional modules. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3653–3667, 2022

    2022

  38. [47]

    Preserving in-context learning ability in large language model fine-tuning

    Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit S Dhillon, and Sanjiv Kumar. Preserving in-context learning ability in large language model fine-tuning. 2022

    2022

  39. [48]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proceedings of the Eleventh International Conference on Learning Representations, ICLR, 2023

    2023

  40. [49]

    Gradient projection memory for continual learning

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In Proceedings of the International Conference on Learning Representations, ICLR, 2021

    2021

  41. [50]

    Visualsimpleqa: A benchmark for decoupled evaluation of large vision-language models in fact-seeking question answering, 2025

    Yanling Wang, Yihan Zhao, Xiaodong Chen, Shasha Guo, Lixin Liu, Haoyang Li, Yong Xiao, Jing Zhang, Qi Li, and Ke Xu. Visualsimpleqa: A benchmark for decoupled evaluation of large vision-language models in fact-seeking question answering, 2025

    2025

  42. [51]

    Binary codes capable of correcting deletions, insertions, and reversals

    Vladimir I Levenshtein et al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union, 1966

    1966

  43. [52]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  44. [53]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  45. [54]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  46. [55]

    Ai2d-rst: a multimodal corpus of 1000 primary school science diagrams

    Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: a multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661–688, 2021

    2021

  47. [56]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  48. [57]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024

    2024

  49. [58]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  50. [59]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024. 12

    2024

  51. [60]

    Author Information\

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 13 A Technical Appendices and Supplementary Material A.1 Training Details...

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.