Pith. sign in

REVIEW 5 major objections 5 minor 41 references

Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large multimodal models are far from solving complex logical reasoning problems that require reading and interpreting text inside images, with the best model scoring 68.1% on a new 150-question benchmark.

desk verdict A genuinely new evaluation probe for complex OCR-cued reasoning, but the per-category claims rest on a small, unvalidated sample and should be treated as tentative. read the letter →

arxiv 2505.12766 v1 pith:GEFEK44W submitted 2025-05-19 cs.CV

Haibin He , Maoyuan Ye , Jing Zhang , Xiantao Cai , Juhua Liu , Bo Du , Dacheng Tao This is my paper · ORCID
classification cs.CV
keywords largemultimodalmodelsopticalcharacterrecognitionlogicalreasoningbenchmarkchain-of-thoughtdecisionvisualquestionansweringOCRcues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Reasoning-OCR, a benchmark of 150 hand-designed logical reasoning questions built from 140 text-rich images across six visual scenarios. The central claim is that current large multimodal models (LMMs) are far from solving these problems: the best proprietary model, GPT-4o, reaches 68.1 percent accuracy, and the best open-source model reaches 63.0 percent. The benchmark is designed to isolate reasoning from OCR cues rather than specialized knowledge, so the low scores point to a genuine deficit in multi-step logical inference over text found in images. This matters because reading text in images is a core skill for document understanding and embodied agents, and existing OCR benchmarks have largely saturated.

What carries the argument

Reasoning-OCR itself is the central object: a benchmark of 150 expert-written questions over 140 images drawn from ChartQA, DocVQA, DT-VQA, and the web. Each question requires extracting textual cues from the image and combining them through multiple inference steps; the six labeled reasoning types are data comparison analysis, data statistical analysis, mathematical reasoning, conditional reasoning, temporal reasoning, and decision reasoning. The benchmark also includes Chinese question versions, a per-question hint, and a task-specific instruction template, which are used to probe whether performance changes when the model is given linguistic scaffolding. The design forces a model to do more than read: it must decide which textual cues are relevant, relate them to the question's constraints, and perform a multi-hop logical chain.

What would settle it

Re-annotate all 150 questions with at least two independent annotators and measure agreement; then add, say, 50 new decision-reasoning questions and rerun the same models. If agreement is low, or if the best model exceeds 50% on the larger decision-reasoning set, the paper's main conclusions about ambiguity and the decision-reasoning bottleneck would need revision.

Watch

Extended reading notes

Core claim

The paper's central discovery is that reasoning over OCR cues is a largely unsolved capability for LMMs. On the 150-question Reasoning-OCR benchmark, GPT-4o achieves 68.1 percent accuracy with chain-of-thought, InternVL2.5-38B achieves 63.0 percent, and most open-source models stay below 50 percent. The benchmark categorizes questions into six reasoning types; the hardest is decision reasoning, where the best models reach only 36.4 percent. The paper also finds that text-centric models specifically trained for OCR lag far behind generic LMMs, that chain-of-thought consistently helps, and that giving a hint improves accuracy. These results are presented as evidence that current OCR benchmarks are saturated and that complex logical reasoning from textual cues remains a bottleneck.

Load-bearing premise

The load-bearing premise is that every question in the benchmark is unambiguous and correctly labeled, and that the per-category sample sizes, especially the roughly eleven decision-reasoning questions, are large enough to support claims about which reasoning types are hardest.

Editorial extensions

If this is right

  • Existing OCR benchmarks that report near-saturated scores may overstate progress, since the same models score below 50 percent on Reasoning-OCR.
  • LMMs are not yet reliable for decision-making tasks that depend on reading text, such as planning and embodied-agent instructions.
  • Chain-of-thought prompting, answer hints, and task-specific instructions each raise accuracy, so test-time scaffolding is a practical lever for improving OCR-based reasoning.
  • Text-centric OCR models should be trained with reasoning-heavy samples, not just recognition and simple visual question answering data.
  • The six-category breakdown offers a diagnostic: statistical and comparison reasoning are relatively stronger, while decision reasoning is weakest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because only about 11 of the 150 questions are decision-reasoning items, the 'decision reasoning is hardest' ranking carries wide error bars; a larger decision-reasoning subset could change the ordering.
  • A natural testable extension is to compare LMMs against a pipeline that first extracts text with an OCR model and then performs reasoning in a pure-language model, which would separate perception errors from reasoning errors.
  • The benchmark's cross-linguistic condition hints at a broader question of whether LMMs reason differently when the same logic is posed in a different language, though the current sample size is too small to resolve it.
  • Releasing per-category confidence intervals and inter-annotator agreement would make the benchmark more useful for tracking progress over time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Reasoning-OCR, a benchmark of 150 bilingual (English/Chinese) questions over 140 text-rich images drawn from ChartQA, DT-VQA, DocVQA, and the web, covering six visual scenarios and six reasoning types (data comparison, data statistics, mathematical, conditional, temporal, and decision reasoning). The authors evaluate ten large multimodal models, including GPT-4o and nine open-source models, under five answering settings (CoT, no-CoT, cross-linguistic, with hint, and task-specific instruction). They report that all models are far from ceiling, that text-centric LMMs lag behind generic LMMs, that decision reasoning is the hardest category, and that CoT and provided hints generally improve accuracy. The benchmark and code are promised to be publicly released.

Significance. If the benchmark is validated, it fills a real gap: existing OCR and visual-text benchmarks mostly test text extraction and simple VQA rather than multi-hop logical reasoning from OCR cues. The design choices are sensible, including diverse scenarios, six explicit reasoning categories, bilingual questions, and multiple inference settings. The qualitative conclusion that current LMMs are far from ceiling on such questions is plausible and would be useful to the community. The paper also provides a detailed error analysis and makes the benchmark, instructions, and code available, which are concrete strengths. However, the quantitative and comparative claims currently rest on 150 unvalidated questions with no inter-annotator agreement, no human accuracy baseline, and no confidence intervals; several internal inconsistencies further undermine the reliability of the reported numbers. The central qualitative finding is likely robust, but the finer claims about relative difficulty across reasoning types and about text-centric versus generic models need additional support.

major comments (5)
  1. [§3.1 and D.3] The benchmark validation is not reported. Section 3.1 asserts that all questions are objective-type with answers that are 'concise and unambiguous,' but the datasheet (D.3, A4) states only that the data were 'collected and verified by the authors.' There is no inter-annotator agreement, no independent annotation study, no pilot validation, and no human accuracy baseline. This matters because every accuracy number in Tables 2 and 3 depends on the assumption that the intended answer is the only defensible one. Without a human ceiling, low model accuracy could partly mean the questions are hard for humans rather than that LMMs are deficient. I request a human evaluation on the full set (or a justified sample), a report of ambiguous or multi-answer questions, and a discussion of how synonymity and format variations were handled.
  2. [§4.3, Finding ❸, Fig. 3(a), Table 2] The claim that decision reasoning is the hardest category rests on about 11 questions (the 0.073 proportion in Fig. 3(a) corresponds to ~11 of 150). In Table 2, GPT-4o's best accuracy of 36.4% is 4/11, and the 95% Wilson interval for 4/11 spans roughly 15%-65%, overlapping the intervals for models scoring 9.1% (1/11) and 18.2% (2/11). One ambiguous or mislabeled decision question changes the category accuracy by about 9 percentage points, which can flip the ranking of models and even the conclusion that decision reasoning is the most difficult. The paper should report confidence intervals, increase the number of decision-reasoning questions, or substantially soften Finding ❸.
  3. [Fig. 1, Figs. 9–10] There are internal annotation inconsistencies that call the data quality into question. In Fig. 1, the English question says 'early March 2023' while the Chinese version says '2003年3月初' (early March 2003); since the evaluation includes cross-linguistic reasoning, a date discrepancy can change the correct answer. Additionally, Fig. 9 is captioned 'An example for mathematical reasoning' but its question is a temporal-reasoning train-ticket problem, and Fig. 10 is captioned 'An example for temporal reasoning' but contains a mathematical expense-sum question. These issues suggest that the bilingual questions and the type labels were not carefully audited, which is load-bearing for a benchmark whose purpose is precise evaluation. The authors should correct these errors and describe a systematic consistency check for all 150 items.
  4. [§4.1] GPT-4o is used as the answer extractor for all models, including GPT-4o itself. The paper follows prior work in doing this, but no evidence is provided that the extraction is unbiased or accurate. If GPT-4o is more lenient toward answers that match its own output format or if it silently normalizes incorrect answers, then the reported accuracies of open-source models could be affected, and GPT-4o's comparative advantage could be inflated. The authors should either measure extractor agreement against human judgments, use rule-based/string matching combined with a fixed normalization step, or report extraction-error statistics.
  5. [§4.3, Finding ❷, Table 2] The comparison between text-centric and generic LMMs is confounded. TextMonkey and mPLUG-DocOwl2 differ from Qwen2-VL-7B and InternVL2.5-8B not only in training specialization but also in base architecture, parameter count, and training data scale. The claim that text-centric training itself limits reasoning ability is not established by these comparisons. I suggest either adding matched-scale text-centric and generic models with comparable base backbones or recasting Finding ❷ as an observation about the specific evaluated models rather than about text-centric training as a general principle.
minor comments (5)
  1. [Abstract and §3.2] The abstract contains 'underscoring the urgent to improve the reasoning performance,' which appears to be missing a noun ('urgent need'), and §3.2 says 'Expect the questions in English' where 'Except' is intended. Please proofread these passages.
  2. [Table 2] The column headers 'Datac', 'Datas', 'Reasoning m', 'Reasoning c', 'Reasoning t', and 'Reasoning d' are hard to parse. Please define the abbreviations in the table caption or use the full names of the six reasoning categories.
  3. [Table 3] The symbols ACC, ACCn, ACCl, ACCh, and ACCt are clear from the caption, but the notation should be introduced in text right before the table for readability.
  4. [Limitations] The Limitations section discusses scaling and scenario breadth but does not mention the absence of validation, inter-annotator agreement, or human baselines. Adding a sentence acknowledging these limitations would help readers calibrate the claims.
  5. [Fig. 5] The GPT-4o response in the 'Question Misunderstanding' example does not follow the required concise <a>...</a> format. Since the paper uses GPT-4o as the answer extractor, this example makes it particularly important to report how such verbose responses were scored.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark and its findings are empirical measurements, with no fitted parameters or self-citation chain that reduces to its own inputs.

full rationale

Reasoning-OCR is a benchmark-construction and evaluation paper. The central claims are measured accuracies of LMMs on newly written questions, not quantities derived from equations or from fitted parameters. Benchmark construction (selecting images, writing questions, annotating answers) and evaluation (computing accuracy from model outputs against the annotated answers) are separate operations; no output is equivalent to an input by construction. The use of GPT-4o both as an evaluated model and as an answer extractor is a potential measurement-bias concern, but it is not a logical circularity: the extraction step does not define the ground-truth answers or numerically force the reported accuracies. No load-bearing self-citation chain appears; the source datasets (ChartQA, DT-VQA, DocVQA) are external and the benchmark's new questions are authored for this paper. The absence of inter-annotator agreement, human-ceiling measurement, or confidence intervals is a validity and robustness concern about the strength of comparative findings such as 'decision reasoning is hardest,' but it is not a circularity in the derivation. Under the required standard of exhibiting a specific reduction (e.g., Eq. X = Eq. Y by definition, or a fitted parameter renamed as a prediction), no circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's measurements rest on three assumptions that are not independently verified: the ground-truth answers are correct and unambiguous, the GPT-4o answer extractor is unbiased across all evaluated models, and the source images have not been memorized by the evaluated models. These are not fitted parameters or invented entities; they are domain assumptions that a user must accept to trust the reported accuracy.

assumptions (4)
  • domain assumption Each of the 150 questions has exactly one correct, unambiguous answer, as judged by the authors.
    The benchmark's accuracy metric treats all ground-truth answers as definitive. No inter-annotator agreement, second-opinion check, or pilot study is reported in Sec. 3.1 or the datasheet.
  • domain assumption GPT-4o is a reliable and unbiased answer extractor for all evaluated models.
    Sec. 4.1 uses GPT-4o to extract final answers from verbose outputs. GPT-4o is also one of the evaluated models, so its own answers pass through itself, and open-source models' formats may be handled differently.
  • domain assumption The evaluated LMMs have not memorized the source images (ChartQA, DocVQA, DT-VQA) or their associated question-answer pairs during training.
    60 of 140 images come from ChartQA and 20 from DocVQA, widely used public benchmarks. No contamination check is mentioned in Sec. 3.1 or Sec. 4.
  • domain assumption The six reasoning categories are mutually exclusive and consistently applied.
    Fig. 3 reports per-category distributions, but the paper does not report annotation guidelines or agreement rates (Sec. 3.2). Some example questions (e.g., the train-ticket one) are labeled 'temporal' in one figure and 'mathematical' in another, suggesting boundary ambiguity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?." pith.science (2026). https://pith.science/paper/GEFEK44W

@misc{pith2026250512766,
  author       = {Pith},
  title        = {Pith review of: Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEFEK44W}},
  note         = {Machine review of arXiv:2505.12766}
}
read the original abstract

Large Multimodal Models (LMMs) have become increasingly versatile, accompanied by impressive Optical Character Recognition (OCR) related capabilities. Existing OCR-related benchmarks emphasize evaluating LMMs' abilities of relatively simple visual question answering, visual-text parsing, etc. However, the extent to which LMMs can deal with complex logical reasoning problems based on OCR cues is relatively unexplored. To this end, we introduce the Reasoning-OCR benchmark, which challenges LMMs to solve complex reasoning problems based on the cues that can be extracted from rich visual-text. Reasoning-OCR covers six visual scenarios and encompasses 150 meticulously designed questions categorized into six reasoning challenges. Additionally, Reasoning-OCR minimizes the impact of field-specialized knowledge. Our evaluation offers some insights for proprietary and open-source LMMs in different reasoning challenges, underscoring the urgent to improve the reasoning performance. We hope Reasoning-OCR can inspire and facilitate future research on enhancing complex reasoning ability based on OCR cues. Reasoning-OCR is publicly available at https://github.com/Hxyz-123/ReasoningOCR.

Figures

Figures reproduced from arXiv: 2505.12766 by the authors.

Figure 1
Figure 1. Samples in Reasoning-OCR. Relevant textual cues in the image are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. Our questions demonstrate higher reasoning complexity compared to the ones in source datasets. plex logical reasoning capabilities using OCR cues. ❷ Text￾centric LMMs lag far behind advanced generic LMMs on rea￾soning. ❸ LMMs fall short in decision reasoning. ❹ CoT g… view at source ↗
Figure 2
Figure 2. The visual scenario distribution in Reasoning-OCR. The collected images cover six visual scenarios, including chart, product label, document, natural image, screen shot, and token (from the most to the least). screen shot, tokens, etc. In addition, compared to mathemat￾ical benchmarks, our benchmark focuses on complex reason￾ing within text-rich images while minimizing the impact of prior and specialized knowledge, … view at source ↗
Figure 3
Figure 3. The distribution of question types. (a) shows the distribution of the six question types across the Reasoning-OCR while (b) describes the distribution of the six question types within each of the data sources. ChartQA Doc-VQA DT-VQA Ours Average 14.2 11.4 11.9 46.1 Max 40 27 36 276 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Visualization of LMMs’ responses on Reasoning-OCR under CoT and task-specific instruction settings. Relevant textual cues in the image are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. The key elements of the question are emphas…
Figure 5
Figure 5. Figure 5: Failure cases of GPT-4o on Reasoning-OCR. Relevant textual cues in the images are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. The key elements in questions are emphasized in blue. In LMMs’ responses, incorrect reasoning steps …
Figure 6
Figure 6. Figure 6: Full instructions under different answering settings. Question in Reasoning-OCR: Considering all the relevant data and details provided in the table, which specific category or type of anesthesiologist appears to demonstrate a higher level of professionalism overall? C…
Figure 7
Figure 7. Figure 7: An example for data comparison analysis. wise rely on external resources (e.g., websites, tweets, other datasets)? A9: The dataset is self-contained. 10. Does the dataset contain data that might be considered [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: An example for data statistical analysis. Question in Reasoning-OCR: The train ticket shown below indicates that the train is scheduled to depart from HYERES, but in reality, it departs 10 minutes earlier. Additionally, the train arrives at Paris GARE LYON 10 minutes l…
Figure 9
Figure 9. Figure 9: An example for mathematical reasoning. Relevant textual cues in the image are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data…
Figure 10
Figure 10. Figure 10: An example for temporal reasoning. Relevant textual cues in the image are highlighted with red circles or rectangles for clarity, which are not visible to LMMs. Chinese question in Reasoning-OCR: 如果彼得被编程为一个旨在执行任务的家用机器人，有一天它会收到主人的特定命令，用微波炉加热一些食物。这项 任务包括分两步加热食物。第一步要求Pet…
Figure 11
Figure 11. Figure 11: An example for conditional reasoning. 2. What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human cu￾ration, software program, software API)? How were these mechanisms or procedures validated? A2: Please refer to th…
Figure 12
Figure 12. Figure 12: An example for decision reasoning. D.4 Preprocessing/cleaning/labeling 1. Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, pro￾cessing 5 of…
Figure 13
Figure 13. Figure 13: Examples of generating complex reasoning questions based on OCR cues in images using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 18 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , volume 35, pages 23716--23736, 2022

    work page 2022

  3. [3]

    Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. In Findings of ACL-IJCNLP , pages 513--523, 2021

    work page 2021

  4. [4]

    Onechart: Purify the chart structural extraction via one auxiliary token

    Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, et al. Onechart: Purify the chart structural extraction via one auxiliary token. In ACM MM , pages 147--155, 2024

    work page 2024

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 , 2024

    arXiv 2024

  6. [6]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR , pages 24185--24198, 2024

    work page 2024

  7. [7]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.06500 , 2, 2023

    arXiv 2023

  8. [8]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008 , 2024

    arXiv 2024

Show all 41 references
  1. [9]

    mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, et al. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. arXiv preprint arXiv:2409.03420 , 2024

    2024 arXiv

  2. [11]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024

    2024 arXiv

  3. [12]

    Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790 , 2024

    2024 arXiv

  4. [13]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, et al. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR , pages 26763--26773, 2024

    2024

  5. [14]

    Focus anywhere for fine-grained multi-page document understanding

    Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Focus anywhere for fine-grained multi-page document understanding. arXiv preprint arXiv:2405.14295 , 2024

    2024 arXiv

  6. [15]

    Mmc: Advancing multimodal chart understanding with large-scale instruction tuning

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, et al. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. In NAACL , pages 1287--1310, 2024

    2024

  7. [16]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR , pages 26296--26306, 2024

    2024

  8. [17]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  9. [18]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, et al. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences , 67(12):220102, 2024

    2024

  10. [19]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, et al. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 , 2024

    2024 arXiv

  11. [20]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR , 2024

    2024

  12. [21]

    Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms

    Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352 , 2024

    2024 arXiv

  13. [22]

    Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code

    Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code. arXiv preprint arXiv:2410.08196 , 2024

    arXiv

  14. [23]

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. In NeurIPS Datasets and Benchmarks Track , 2024

    2024

  15. [24]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL , pages 2263--2279, 2022

    2022

  16. [25]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV , pages 2200--2209, 2021

    2021

  17. [26]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  18. [27]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, et al. Towards vqa models that can read. In CVPR , pages 8317--8326, 2019

    2019

  19. [28]

    Mtvqa: Benchmarking multilingual text-centric visual question answering

    Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering. arXiv preprint arXiv:2405.11985 , 2024

    2024 arXiv

  20. [29]

    Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models

    Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models. In ICML , 2024

    2024

  21. [30]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804 , 2024

    2024 arXiv

  22. [31]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024

    2024 arXiv

  23. [32]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. In NeurIPS Datasets and Benchmarks Track , 2024

    2024

  24. [33]

    General ocr theory: Towards ocr-2.0 via a unified end-to-end model

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. arXiv preprint arXiv:2409.01704 , 2024

    2024 arXiv

  25. [34]

    Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning

    Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, et al. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185 , 2024

    2024 arXiv

  26. [35]

    Chartbench: A benchmark for complex visual reasoning in charts

    Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915 , 2023

    2023 arXiv

  27. [36]

    If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents

    Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, et al. If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. arXiv preprint arXiv:2401.008...

    2024 arXiv

  28. [37]

    Cc-ocr: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy

    Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, et al. Cc-ocr: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy. arXiv preprint arXiv:2412.02210 , 2024

    2024 arXiv

  29. [38]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of EMNLP , pages 2841--2858, 2023

    2023

  30. [39]

    Exploring the capabilities of large multimodal models on dense text

    Shuo Zhang, Biao Yang, Zhang Li, Zhiyin Ma, Yuliang Liu, and Xiang Bai. Exploring the capabilities of large multimodal models on dense text. In ICDAR , pages 281--298, 2024

    2024

  31. [40]

    Unveiling the impact of coding data instruction fine-tuning on large language models reasoning

    Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. arXiv preprint arXiv:2405.20535 , 2024

    2024 arXiv

  32. [41]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In ECCV , pages 169--186, 2025

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In ECCV , pages 169--186, 2025

    2025

  33. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.