Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3B-parameter open model matches much larger systems at reading documents.

desk verdict A real, useful open 3B document-VLM; trust the absolute scores, but the 'matches GPT-4o/Gemini' headline rests on numbers from different pipelines. read the letter →

arxiv 2502.09927 v1 pith:6YGP4SWU submitted 2025-02-14 cs.CV cs.AI

Granite Vision Team: Leonid Karlinsky , Assaf Arbelle , Abraham Daniels , Ahmed Nassar , Amit Alfassi , Bo Wu , Eli Schwartz , Dhiraj Joshi
show 54 more authors
Jovana Kondic Nimrod Shabtay Pengyuan Li Roei Herzig Shafiq Abedin Shaked Perek Sivan Harary Udi Barzelay Adi Raz Goldfarb Aude Oliva Ben Wieles Bishwaranjan Bhattacharjee Brandon Huang Christoph Auer Dan Gutfreund David Beymer David Wood Hilde Kuehne Jacob Hansen Joseph Shtok Ken Wong Luis Angel Bathen Mayank Mishra Maksym Lysak Michele Dolfi Mikhail Yurochkin Nikolaos Livathinos Nimrod Harel Ophir Azulai Oshri Naparstek Rafael Teixeira de Lima Rameswar Panda Sivan Doveh Shubham Gupta Subhro Das Syed Zawad Yusik Kim Zexue He Alexander Brooks Gabe Goodhart Anita Govindjee Derek Leist Ibrahim Ibrahim Aya Soffer David Cox Kate Soule Luis Lastras Nirmit Desai Shila Ofek-koifman Sriram Raghavan Tanveer Syeda-Mahmood Peter Staar Tal Drory Rogerio Feris
This is my paper
classification cs.CVcs.AI
keywords GraniteVisionvision-languagemodelvisualdocumentunderstandingenterpriseAIchartandtableextractionLiveXivbenchmarksparseattentionvectorsApache-2.0open
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Granite Vision is a vision-language model of about 3 billion parameters, built for enterprise document understanding: reading tables, charts, forms, receipts, and diagrams. The paper argues that lightweight models need not concede this domain to much larger systems. On standard benchmarks it reports leading scores among 1-4B models—DocVQA 0.88, ChartQA 0.86, OCRBench 0.75, LiveXiv TQA 0.55—and says it matches or exceeds several 7B-90B and proprietary models on document tasks. The model is released under Apache-2 with visibility into training data, and the authors add a test-time safety classifier based on sparse attention vectors. Readers should care because the claim, if true, means strong document AI can run at a fraction of the compute and licensing cost of frontier models.

What carries the argument

The load-bearing machinery has two parts. The first is the vision-language alignment stack: a SigLIP encoder at 384x384 resolution, an AnyRes tiling scheme that splits an image into up to ten 384x384 patches across 27 aspect-ratio choices, a two-layer MLP projector, and the Granite 3.1-2B decoder-only language model; the projector is trained first, then the projector plus language model in a second pre-training stage, then full instruction tuning on roughly 20 million image-text pairs. The second is the Sparse Attention Vectors (SAVs) method: from a few labeled examples it scores every attention head by how well a nearest-class-centroid cosine classifier separates safety classes, keeps the top-K heads, and classifies new inputs by majority vote among those heads.

What would settle it

Re-evaluate every model in Table 1 under identical prompting, decoding, and post-processing conditions, and if several larger models then outscore Granite Vision on DocVQA or ChartQA by a wide margin, the central comparison fails; an independent check for training and test overlap with ChartQA and DocVQA would also settle whether contamination explains the scores.

Watch

Extended reading notes

Core claim

The paper's central claim is that a surprisingly small model can be made expert at reading documents by combining the right base language model, a high-resolution multi-patch vision encoder, and an enormous curated instruction-following dataset. Granite Vision—about 3B parameters in total—scores 0.88 on DocVQA, 0.86 on ChartQA, 0.90 on WebSRC, and 0.55 on the contamination-resistant LiveXiv table QA benchmark, numbers the authors report as leading among 1-4B models and competitive with 7-90B and proprietary models on document-related tasks. The same model also converts table and chart images to HTML or Markdown with TEDS and mTEDS scores close to a 12B model, and its sparse-attention-vector safety classifier raises accuracy on VLGuard from 86.0 to 96.2. The authors present Granite Vision as evidence that careful data curation and training procedure, not parameter count, are what move document understanding.

Load-bearing premise

The comparison to larger models rests on the assumption that the benchmark numbers in Table 1 are directly comparable, even though the authors ran only their own model in a controlled setting and took the starred models' scores from original publications.

Editorial extensions

If this is right

  • A 3B open-weight model can be deployed for document question answering and structured extraction at compute costs far below 7B-parameter and proprietary systems.
  • On table and chart extraction, Granite Vision's scores are close to a model four times its size, suggesting the capability transfers to automated structured-data pipelines.
  • The Apache-2 release with training-data visibility allows commercial adoption and auditability that closed or restricted models do not offer.
  • The contamination-resistant LiveXiv evaluation makes the document-understanding results less likely to be explained by memorization of web-scraped test images.
  • Adding Sparse Attention Vectors improves safety classification accuracy, indicating that the generative model already contains discriminative safety features that can be surfaced at test time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The training recipe suggests that synthetic data built from verbalized documents, augmented chart tables, and flowchart templates may be the dominant factor, implying similar recipes could lift other small models on document-heavy tasks without extra parameters.
  • The SAV classifier could be detached from Granite Vision and reused as a lightweight safety filter in front of other multimodal systems; one testable extension is whether the same selected heads transfer to new safety categories beyond VLGuard and MHalu.
  • The mTEDS chart-extraction results imply that small models may be viable for automated data-wrangling on financial and scientific charts; a natural next test is noisy real-world scans where OCR-based pipelines typically fail.
  • Because LiveXiv is continuously updated, the benchmark itself offers a test of whether Granite Vision's margin persists as new papers and new questions enter the evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces Granite Vision, a roughly 3-billion-parameter vision-language model built on the Granite 3.1-2B language model with a SigLIP vision encoder and an MLP projector. The model is trained in three stages on a large instruction-following dataset that combines synthetic data generated from the IBM DocFM document corpus with many public document and general-image datasets. The reported evaluation covers standard document-understanding and general VQA benchmarks, table/chart extraction tasks, the LiveXiv contamination-resistant benchmark, and a safety classification method based on sparse attention vectors (SAVs). The model weights are released under the Apache-2 license.

Significance. If the reported results are substantiated, Granite Vision would be a valuable lightweight open document-understanding model: it beats several 4B models on ChartQA (0.86 vs 0.82 for Phi-3.5v) and OCRBench (0.75 vs 0.64), and it approaches the performance of much larger models on several tasks. The paper's strengths include the use of the standardized lmms-eval harness for the authors' own runs, the use of LiveXiv to mitigate test contamination, the detailed description of the synthetic data pipeline, and the public release of model weights. However, the comparative claims are weakened by the heterogeneous evaluation protocols in Table 1 and by the lack of error bars, so the precise extent of the improvement over prior open models is currently uncertain.

major comments (4)
  1. [Section 5.1, Table 1] The central comparative claims — 'state-of-the-art results' in the Introduction and 'competitive with much larger proprietary models' in Section 5.1 — rest on Table 1, which mixes the authors' controlled lmms-eval runs with numbers marked with an asterisk that were taken from other publications or public benchmarks. The footnote explicitly states that those starred models were not run in the controlled setup. Since prompts, decoding settings, answer post-processing, and even metric definitions (e.g., ANLS vs accuracy for DocVQA) vary across sources, the apparent near-parity with GPT-4o on ChartQA (0.86 vs 0.85) and the ordinal comparisons against MM1.5, SmolVLM, Cambrian-1, and others are not established. To support the claims, either re-run all comparison models in the same harness or restrict the comparative statements to the models that were actually run in-house.
  2. [Section 5.1, Table 1 and Section 2.1] The 'state-of-the-art in its size category' claim is also vulnerable to an incomplete comparison set. Table 1 lists only five small (1B-4B) competitors and omits several widely used open models of that size, such as Qwen2-VL-2B, InternVL2-2B, and MiniCPM-V 2.6, which have strong document-understanding results. Without these baselines, the claim that Granite Vision achieves state-of-the-art among 1B-4B models is not supported. The authors should add these models or justify their absence.
  3. [Section 5.1, Tables 1-4] No error bars, confidence intervals, or number of random seeds are reported for any of the quantitative results. Several differences that drive the narrative, such as the DocVQA tie at 0.88 with Phi-3.5v and the LiveXiv VQA gap of 0.61 vs 0.63, are small enough to be within run-to-run variability. This is especially important for the safety classification results in Table 4, where the improvement from adding Safety Vectors on VLGuard (86.0 to 96.2) is large but reported without any variance estimate. Please report multiple seeds or a bootstrap interval for the headline comparisons.
  4. [Section 4.3 and Table 4] The SAV safety classification method depends on two free parameters: K, the number of top-scoring attention heads retained, and N, the number of few-shot samples used to compute centroids. Neither value is reported for the results in Table 4, and no procedure is given for choosing them. Without this information the method is not reproducible, and the reader cannot rule out that the reported 80.7 on MHalu and 96.2 on VLGuard were obtained by tuning these parameters on the test set. Please specify K, N, the selection criterion, and any validation protocol.
minor comments (5)
  1. [Section 3.1.2] The text says DocFM-VQA contains 'a total of 20 million QA pairs' while Figure 2 lists 19.9M QA pairs; please reconcile the two numbers.
  2. [Section 4.2, Stage 3] Stage 3 says the model is trained on 'approximately 20M image-text pairs,' while the Introduction states the dataset has '80 million instructions.' Please clarify the relationship between instructions and image-text pairs.
  3. [Section 5.1, Table 1 caption] The footnote distinguishing in-house lmms-eval results from numbers taken from other papers is easy to miss and would be more transparent if the table had a separate column for the evaluation source or a more prominent typographic mark.
  4. [Section 5.3, Table 3] The column header 'Politic' should be 'Politics' for consistency with the benchmark name RTVLM.
  5. [Section 5.1, LiveXiv results] The phrasing 'strong results on the LiveXiv benchmark' is too strong when read against Table 1: the LiveXiv VQA score (0.61) is below Phi-3.5v (0.63), although the TQA score (0.55) is the best among the small models. Please qualify the claim accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark claims rest on external evaluations, not on self-defined quantities.

full rationale

The paper's central claim is that a roughly 3B-parameter model achieves strong scores on standard document-understanding benchmarks and on LiveXiv. These scores are direct empirical measurements on external test sets (DocVQA, ChartQA, TextVQA, AI2D, InfoVQA, OCRBench, WebSRC, LiveXiv, MMMU, VQAv2, RealWorldQA, VizWiz, OK-VQA) run through the lmms-eval harness; the derivation chain is training-data construction, model training, then held-out evaluation. No reported result is obtained by substituting the paper's own definitions into an equation, and no fitted parameter is renamed as a prediction. The safety-classification method selects sparse attention heads using scores computed on few-shot training samples and then evaluates on held-out test data, which is a standard train/test split rather than a construction-level circularity. Self-citations (Granite LLM, Docling, DocFM, TableFormer, LiveXiv, Sparse Attention Vectors) are used as tools, infrastructure, or benchmark assets; none is the sole justification for the benchmark outcomes, and LiveXiv's content is externally sourced from recently published arXiv papers rather than derived from the model's own outputs. Table 1's footnote transparently discloses that some comparison numbers were taken from original publications or public benchmarks instead of being rerun in the controlled lmms-eval setup; this is an evaluation-comparability limitation, not a circular reduction. The mTEDS metric is an explicitly stated modification of TEDS applied post-hoc to extraction outputs, not a quantity optimized to match the reported scores. In summary, every load-bearing claim is empirically checkable against external data, and no step reduces by construction to the paper's own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The model's performance is an empirical measurement, so the main burdens are evaluation fairness and data validity. The paper does not release the training code or the DocFM synthetic datasets, and it relies on heterogeneous benchmark numbers for the claimed state-of-the-art. The SAV safety classifier additionally depends on two unreported hyperparameters (K and N).

free parameters (2)
  • SAV top-K head count
    Section 4.3 Step 2 selects the K top-scoring attention heads for safety classification; K is never reported. Table 4's 96.2% VLGuard score depends on this choice, which is fitted to the few-shot training set.
  • SAV few-shot sample count N
    Step 1 uses N few-shot examples per class to build centroids and rank heads. N is unspecified, and the classification results in Table 4 are sensitive to this sample size.
assumptions (3)
  • domain assumption Test splits of DocVQA, ChartQA, TextVQA, AI2D, InfoVQA and other standard benchmarks do not appear in the DocFM web-crawled corpus or other training data.
    The paper uses LiveXiv precisely to avoid contamination, but does not check whether standard benchmark pages leaked into the 4.6M PDFs crawled from Common Crawl (Section 3.1.1).
  • domain assumption Benchmark numbers from different sources are directly comparable.
    Table 1 mixes controlled lmms-eval runs with results marked * from original publications, without correcting for prompt or decoding differences.
  • domain assumption Synthetic QA pairs generated from verbalized document text teach the model to answer visual questions.
    DocFM-VQA (Section 3.1.2) generates questions from text and verbal descriptions of charts/tables; some questions may be answerable from language priors alone, and the authors note they did not filter hallucinations in generated QAs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence." pith.science (2026). https://pith.science/paper/6YGP4SWU

@misc{pith2026250209927,
  author       = {Pith},
  title        = {Pith review of: Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YGP4SWU}},
  note         = {Machine review of arXiv:2502.09927}
}
read the original abstract

We introduce Granite Vision, a lightweight large language model with vision capabilities, specifically designed to excel in enterprise use cases, particularly in visual document understanding. Our model is trained on a comprehensive instruction-following dataset, including document-related tasks, such as content extraction from tables, charts, diagrams, sketches, and infographics, as well as general image tasks. The architecture of Granite Vision is centered around visual modality alignment with a decoder-only, 2 billion parameter Granite large language model. Additionally, we introduce a dedicated safety classification approach in test-time that leverages a sparse set of attention vectors to identify potential harmful inputs. Despite its lightweight architecture, Granite Vision achieves strong results in standard benchmarks related to visual document understanding, as well as on the LiveXiv benchmark, which is designed to avoid test set contamination by using a constantly updated corpus of recently published Arxiv papers. We are releasing the model under the Apache-2 license, allowing for both research and commercial use, while offering complete visibility into the training data and other relevant details. See https://huggingface.co/ibm-granite/ for model weights.

Figures

Figures reproduced from arXiv: 2502.09927 by the authors.

Figure 1
Figure 1. Qualitative examples generated by Granite Vision, showcasing its diverse capabilities including (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our comprehensive collection of document understanding datasets used for Granite [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of general image datasets used for Granite Vision training. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Architecture of Granite Vision. Vision Encoder vϕ. For an input image I, we use a vision encoder vϕ to provide visual features X = vϕ(I). In our implementation, X is a con￾catenation of outputs from multiple layers, allowing us to combine these different levels of repr…

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction

    cs.CV 2025-12 conditional novelty 7.0 of 10

    PubTables-v2 is a large annotated dataset for table extraction spanning cropped tables, full pages, and full documents, including the first large benchmark of multi-page tables.

  2. Aligning Quantum Operators with Large Language Models

    quant-ph 2026-06 conditional novelty 6.0 of 10

    An LLM that reads a quantum operator as image-like patches can synthesize 4-qubit Pauli-rotation circuits at high success and obey English gate constraints.

  3. CARES: Context-Aware Resolution Selector for VLMs

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A lightweight selector trained on target-VLM rollouts routes each image-query pair to the minimal sufficient resolution, cutting prefill compute by 63-80% across five benchmarks at roughly matched accuracy.

  4. ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation

    cs.HC 2025-05 conditional novelty 6.0 of 10

    A fully automated pipeline generates a 222.5K-pair synthetic chart dataset with 27 chart types and 11 plotting libraries, and a GPT-4o-judged benchmark shows current open-weights VLMs still underperform on chart-to-co...

  5. Spoken question answering for visual queries

    eess.AS 2025-05 conditional novelty 6.0 of 10

    A LLaVA-style model with an added Whisper speech encoder answers spoken questions about images, trained on TTS-synthesized speech and reaching near the text-input baseline.

  6. Activation Reward Models for Few-Shot Model Alignment

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Mean attention-head activations from a few labeled examples, injected into selected heads, turn a frozen vision-language model into a few-shot reward model that beats prompting and scoring baselines and a new reward-h...

  7. Transcribing Spanish Texts from the Past: Experiments with Transkribus, Tesseract and Granite

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A 2B-parameter multimodal model fine-tuned with QLoRA on a gaming PC beat larger OCR baselines on historical Spanish document transcription, while Tesseract fine-tuning offered no gain.

Reference graph

Works this paper leans on

112 extracted references · 39 canonical work pages · cited by 7 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [3]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024 b

    arXiv 2024

  3. [4]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall,...

    arXiv 2024

  4. [5]

    Flamingo: a visual language model for few-shot learning, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    arXiv 2022

  5. [6]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-01-15

    2024

  6. [7]

    Manmatha

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. Docformer: End-to-end transformer for document understanding, 2021. URL https://arxiv.org/abs/2106.11539

    arXiv 2021

  7. [8]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL https://arxiv.org/abs/2308.12966

    arXiv 2023

  8. [9]

    Automatikz: Text-guided synthesis of scientific vector graphics with tikz, 2023

    Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz, 2023

    2023

Show all 112 references
  1. [10]

    Scene text visual question answering, 2019

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar c al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering, 2019

    2019

  2. [11]

    Due: End-to-end document understanding benchmark, 2021

    ukasz Borchmann, Micha Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Micha Turski, Karolina Szyndler, and Filip Grali \'n ski. Due: End-to-end document understanding benchmark, 2021

    2021

  3. [12]

    Unified hallucination detection for multimodal large language models, 2024 a

    Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multimodal large language models, 2024 a

    2024

  4. [13]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  5. [14]

    Finqa: A dataset of numerical reasoning over financial data, 2021

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data, 2021

    2021

  6. [15]

    Hitab: A hierarchical table dataset for question answering and natural language generation, 2022

    Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation, 2022

    2022

  7. [16]

    Nvlm: Open frontier-class multimodal llms, 2024

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms, 2024. URL https://arxiv.org/abs/2409.11402

    2024 arXiv

  8. [17]

    Smith, Hannaneh Hajishirzi, Ross B

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch...

    2024 doi arXiv

  9. [18]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models, 2024 b

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models, 2024 b

    2024

  10. [19]

    Turl: Table understanding through representation learning, 2022

    Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: Table understanding through representation learning, 2022

    2022

  11. [20]

    Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding, 2024

    Hao Feng, Qi Liu, Hao Liu, Jingqun Tang, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding, 2024. URL https://arxiv.org/abs/2311.11810

    2024 arXiv

  12. [21]

    Wikimedia downloads

    Wikimedia Foundation. Wikimedia downloads. URL https://dumps.wikimedia.org

  13. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    2017

  14. [23]

    Granite 3.0 language models, 2024

    IBM Granite Team. Granite 3.0 language models, 2024

    2024

  15. [24]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and ... The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  16. [25]

    Vizwiz grand challenge: Answering visual questions from blind people, 2018

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018

    2018

  17. [26]

    Evaluation of deep convolutional nets for document image classification and retrieval, 2015

    Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval, 2015

    2015

  18. [27]

    Screenqa: Large-scale question-answer pairs over mobile app screenshots, 2022

    Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots, 2022

    2022

  19. [28]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024 a

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024 a . URL https://arxiv.org/abs/2403.12895

    2024 arXiv

  20. [29]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024 b

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024 b

    2024

  21. [30]

    Multimodal task vectors enable many-shot multimodal in-context learning, 2024

    Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learning, 2024

    2024

  22. [31]

    Layoutlmv3: Pre-training for document ai with unified text and image masking, 2022

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking, 2022. URL https://arxiv.org/abs/2204.08387

    2022 arXiv

  23. [32]

    Smolvlm, 2024

    Hugging-Face. Smolvlm, 2024. URL https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct

    2024

  24. [33]

    Docmatix: A new approach to document understanding, 2025

    HuggingFace. Docmatix: A new approach to document understanding, 2025. URL https://huggingface.co/blog/docmatix. Accessed: 2025-01-14

    2025

  25. [34]

    Dvqa: Understanding data visualizations via question answering, 2018

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering, 2018

    2018

  26. [35]

    Figureqa: An annotated figure dataset for visual reasoning, 2017

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, \'A kos K \'a d \'a r, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2017

    2017

  27. [36]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pp.\ 161-...

    2024

  28. [37]

    A diagram is worth a dozen images, 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016

    2016

  29. [38]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension, 2017

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension, 2017

    2017

  30. [39]

    Ocr-free document understanding transformer, 2022 a

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer, 2022 a

    2022

  31. [40]

    Ocr-free document understanding transformer, 2022 b

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer, 2022 b . URL https://arxiv.org/abs/2111.15664

    2022 arXiv

  32. [41]

    Rfc 9309 robots exclusion protocol, 2022

    M Koster, G Illyes, H Zeller, and L Sassman. Rfc 9309 robots exclusion protocol, 2022

    2022

  33. [42]

    Unlocking the conversion of web screenshots into html code with the websight dataset, 2024

    Hugo Lauren c on, L \'e o Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset, 2024

    2024

  34. [43]

    Building and better understanding vision-language models: insights and future directions., 2024 a

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions., 2024 a

    2024

  35. [44]

    What matters when building vision-language models?, 2024 b

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024 b

    2024

  36. [45]

    Lmms-eval: Accelerating the development of large multimodal models, March 2024 a

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, March 2024 a . URL https://github.com/EvolvingLMMs-Lab/lmms-eval

    2024

  37. [46]

    Llava-onevision: Easy visual task transfer, 2024 b

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024 b

    2024

  38. [47]

    Llava-onevision: Easy visual task transfer, 2024 c

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024 c . URL https://arxiv.org/abs/2408.03326

    2024 arXiv

  39. [48]

    Llava-onevision: Easy visual task transfer, 2024 d

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer, 2024 d

    2024

  40. [49]

    BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    2023

  41. [50]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024 e

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024 e

    2024

  42. [51]

    Red teaming visual language models, 2024 f

    Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. Red teaming visual language models, 2024 f . URL https://api.semanticscholar.org/CorpusID:267094801

    2024

  43. [52]

    Doclayllm: An efficient and effective multi-modal extension of large language models for text-rich document understanding, 2024

    Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin. Doclayllm: An efficient and effective multi-modal extension of large language models for text-rich document understanding, 2024. URL https://arxiv.org/abs/2408.15045

    2024 arXiv

  44. [53]

    Hrvda: High-resolution visual document assistant, 2024 a

    Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu. Hrvda: High-resolution visual document assistant, 2024 a . URL https://arxiv.org/abs/2404.06918

    2024 arXiv

  45. [54]

    Deplot: One-shot visual language reasoning by plot-to-table translation, 2023 a

    Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One-shot visual language reasoning by plot-to-table translation, 2023 a . URL https://arxiv.org/abs/2212.10505

    2023 arXiv

  46. [55]

    Improved baselines with visual instruction tuning, 2023 b

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023 b

    2023

  47. [56]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023 c

    2023

  48. [57]

    Ocrbench: on the hidden mystery of ocr in large multimodal models, 2024 b

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models, 2024 b

    2024

  49. [58]

    Nikolaos Livathinos, Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Kasper Dinkla, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, and Peter W. J....

    2025 arXiv

  50. [59]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, 2022

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, 2022

    2022

  51. [60]

    Optimized table tokenization for table structure recognition

    Maksym Lysak, Ahmed Nassar, Nikolaos Livathinos, Christoph Auer, and Peter Staar. Optimized table tokenization for table structure recognition. In Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San Jos\' e , CA, USA, August 21–26, 2023, Proceedi...

    2023 doi

  52. [61]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    2019

  53. [62]

    The iam-database: an english sentence database for offline handwriting recognition, 2002

    U-V Marti and Horst Bunke. The iam-database: an english sentence database for offline handwriting recognition, 2002

    2002

  54. [63]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    2022

  55. [64]

    Unichart: A universal vision-language pretrained model for chart comprehension and reasoning, 2023

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning, 2023

    2023

  56. [65]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp.\ 2200--2209, 2021

    2021

  57. [66]

    Infographicvqa, 2022

    Minesh Mathew, Viraj Bagal, Rub \`e n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa, 2022

    2022

  58. [67]

    Plotqa: Reasoning over scientific plots, 2020

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots, 2020

    2020

  59. [68]

    Scene text recognition using higher order language priors, 2012

    Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors, 2012

    2012

  60. [69]

    Ocr-vqa: Visual question answering by reading text in images, 2019

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images, 2019

    2019

  61. [70]

    Sparse attention vectors: Generative multimodal model features are discriminative vision-language classifiers, 2024

    Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Karlinsky, Trevor Darrell, Deva Ramanan, and Roei Herzig. Sparse attention vectors: Generative multimodal model features are discriminative vision-language classifiers, 2024

    2024

  62. [71]

    Kvp10k: A comprehensive dataset for key-value pair extraction in business documents

    Oshri Naparstek, Ophir Azulai, Inbar Shapira, Elad Amrani, Yevgeny Yaroker, Yevgeny Burshtein, Roi Pony, Nadav Rubinstein, Foad Abo Dahood, Orit Prince, et al. Kvp10k: A comprehensive dataset for key-value pair extraction in business documents. In International Conference on D...

    2024

  63. [72]

    Tableformer: Table structure understanding with transformers

    Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. Tableformer: Table structure understanding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 4614--4623, June 2022

    2022

  64. [73]

    Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model, 2020

    Jason Obeid and Enamul Hoque. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model, 2020

    2020

  65. [74]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023 a

    2023 arXiv

  66. [75]

    Gpt-4v(ision) technical work and authors, 2023 b

    OpenAI. Gpt-4v(ision) technical work and authors, 2023 b . URL https://openai.com/contributions/gpt-4v/. Accessed: 2025-01-15

    2023

  67. [76]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/. Accessed: 2025-01-15

    2024

  68. [77]

    Nassar, and Peter Staar

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar. Doclaynet: A large human-annotated dataset for document-layout segmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pp.\ 3743–3751, New...

    2022

  69. [78]

    Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam Laradji, Spandanna Gella, Perouz Taslakian, David Vazquez, and Sai Rajeswar

    Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats Leon Richter, Saverio ...

    2024 arXiv

  70. [79]

    N umero L ogic: Number encoding for enhanced LLM s' numerical reasoning

    Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, and Assaf Arbelle. N umero L ogic: Number encoding for enhanced LLM s' numerical reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical M...

    2024 doi

  71. [80]

    Livexiv--a multi-modal live benchmark based on arxiv papers content, 2024

    Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, M Jehanzeb Mirza, Leshem Chosen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, et al. Livexiv--a multi-modal live benchmark based on arxiv papers content, 2024

    2024

  72. [81]

    Textcaps: a dataset for image captioning with reading comprehension, 2020

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020

    2020

  73. [82]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019

    2019

  74. [83]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text, 2021

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text, 2021

    2021

  75. [84]

    Pubtables-1m: Towards comprehensive table extraction from unstructured documents

    Brandon Smock, Rohith Pesala, and Robin Abraham. Pubtables-1m: Towards comprehensive table extraction from unstructured documents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4634--4642, 2022

    2022

  76. [85]

    Visualmrc: Machine reading comprehension on document images, 2021

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmrc: Machine reading comprehension on document images, 2021

    2021

  77. [86]

    Vistext: A benchmark for semantically rich chart captioning, 2023

    Benny J Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning, 2023

    2023

  78. [87]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530

    2024 arXiv

  79. [88]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024....

    2024 arXiv

  80. [89]

    Ccpdf: Building a high quality corpus for visually rich documents from web crawl data, 2023 a

    Micha Turski, Tomasz Stanis awek, Karol Kaczmarek, Pawe Dyda, and Filip Grali \' n ski. Ccpdf: Building a high quality corpus for visually rich documents from web crawl data, 2023 a

    2023

  81. [90]

    Ccpdf: Building a high quality corpus for visually rich documents from web crawl data, 2023 b

    Micha Turski, Tomasz Stanis awek, Karol Kaczmarek, Pawe Dyda, and Filip Grali \'n ski. Ccpdf: Building a high quality corpus for visually rich documents from web crawl data, 2023 b

    2023

  82. [91]

    Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello

    Shakti N. Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello. The evolution of multimodal model architectures, 2024. URL https://arxiv.org/abs/2405.17927

    2024 arXiv

  83. [92]

    Screen2words: Automatic mobile ui summarization with multimodal learning, 2021

    Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning, 2021

    2021

  84. [93]

    Docllm: A layout-aware generative language model for multimodal document understanding, 2023

    Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. Docllm: A layout-aware generative language model for multimodal document understanding, 2023. URL https://arxiv.org/abs/2401.00908

    2023 arXiv

  85. [94]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024

    2024

  86. [95]

    Tap: Text-aware pre-training for text-vqa and text-caption, 2021

    Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption, 2021

    2021

  87. [96]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model, 2023

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, and Fei Huang. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model, 2023. URL ...

    2023 arXiv

  88. [97]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11 0 (12), November 2024. ISSN 2053-714X. doi:10.1093/nsr/nwae403. URL http://dx.doi.org/10.1093/nsr/nwae403

    2024 doi

  89. [98]

    Syntax-aware network for handwritten mathematical expression recognition, 2022

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition, 2022

    2022

  90. [99]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    2024

  91. [100]

    Implementation and benchmarking of perceptual image hash functions, 2010

    Christoph Zauner. Implementation and benchmarking of perceptual image hash functions, 2010

    2010

  92. [101]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023

    2023

  93. [102]

    Mm1.5: Methods, analysis & insights from multimodal llm fine-tuning, 2024 a

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afsh...

    2024 arXiv

  94. [103]

    Lmms-eval: Reality check on the evaluation of large multimodal models, 2024 b

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024 b . URL https://arxiv.org/abs/2407.12772

    2024 arXiv

  95. [104]

    Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning, 2024 c

    Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning, 2024 c

    2024

  96. [105]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023

    2023

  97. [106]

    Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data, 2022

    Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data, 2022

    2022

  98. [107]

    Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations, 2023

    Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations, 2023

    2023

  99. [108]

    Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context

    Xinyi Zheng, Doug Burdick, Lucian Popa, Peter Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. Winter Conference for Applications in Computer Vision (WACV), 2021

    2021

  100. [109]

    Image-based table recognition: data, model, and evaluation, 2020

    Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. Image-based table recognition: data, model, and evaluation, 2020

    2020

  101. [110]

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models, 2024

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models, 2024

    2024

  102. [111]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  103. [112]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  104. [113]

    2V[ 4ZLOMa?\⎽

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.