Pith. sign in

REVIEW 3 major objections 3 minor 4 cited by

POINTS1.5: Building a Vision-Language Model towards Real World Applications

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read POINTS1.5-7B, trained on under 4 billion tokens, claims first place among sub-10B vision-language models on the OpenCompass leaderboard.

desk verdict Useful training recipe and released weights, but the abstract's token count doesn't match the paper's own table, and the leaderboard claim is thinner than it looks. read the letter →

arxiv 2412.08443 v1 pith:URGVIDG4 submitted 2024-12-11 cs.CV cs.MM

Yuan Liu , Le Tian , Xiao Zhou , Xinyu Gao , Kavio Yu , Yang Yu , Jie Zhou This is my paper · ORCID
classification cs.CVcs.MM
keywords vision-languagemodelNaViTdynamicresolutionbilingualvisualinstructiontuningdatafilteringsoupOpenCompassleaderboard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

POINTS1.5 is a 7-billion-parameter vision-language model built for real-world image tasks such as reading receipts, extracting formulas, describing charts, and solving visual math problems, in both English and Chinese. The paper's central claim is that this model reaches the top of a public multimodal leaderboard among models under 10 billion parameters while being trained on fewer than 4 billion tokens. The authors attribute the result to three changes over their previous model: a vision encoder that natively accepts images at any resolution without splitting them into tiles, newly collected Chinese training data, and systematic filtering of the instruction-tuning data. If the claim holds, a comparatively small training budget can match or beat much larger open models, and the recipe can be reused by other groups.

What carries the argument

The architecture is the conventional LLaVA-style pipeline: a NaViT-style vision encoder (a Vision Transformer that patchifies images of any resolution and aspect ratio into a single patch sequence), a two-layer MLP projector with GELU, and Qwen2.5-7B-Instruct as the language backbone. The carrying mechanism is native dynamic resolution: instead of tiling a large image into fixed-size crops, the encoder processes the whole image and packs multiple image sequences into one long forward pass, recording start and end indices so self-attention stays within each image (via FlashAttention-2). On the data side, CapFusion merges original and VLM-generated captions, perplexity filtering selects the lowest-perplexity portion of the Chinese captions, and instruction data is filtered by LLM-based grammar detection and by discarding answers that are answerable without the image. Model soup averages the best checkpoints to produce the final weights.

What would settle it

Run the released POINTS1.5 checkpoint through the public VLMEvalKit evaluation on the eight leaderboard benchmarks with the same generation settings used by the leaderboard, and compare the reproduced average with the top competing sub-10B models; if another sub-10B model scores higher, or if the reproduced POINTS1.5 score falls below 67.4, the ranking claim is false. Separately, checking whether the token counts in the training logs match the abstract's 'fewer than 4 billion' figure would settle the token-budget sub-claim.

Watch

Extended reading notes

Core claim

The paper claims that POINTS1.5-7B—a LLaVA-style model with a frozen NaViT vision encoder, an MLP projector, and Qwen2.5-7B-Instruct as its language model—achieves the highest OpenCompass average among models with fewer than 10 billion parameters as of December 8, 2024. The claimed result comes from a training budget of about 2.1 billion pretraining tokens plus 2.3 billion instruction-tuning tokens, built from two million captions (roughly half English, half Chinese) and a filtered visual instruction set. The authors report that model soup raises the best single checkpoint from 66.5 to 67.4 on the leaderboard average, and that the model's mathematical performance on MathVista, MATH-Vision, and MathVerse is particularly strong. They also state that filtering out samples whose questions can be answered without the image slightly degraded performance, so those samples were retained.

Load-bearing premise

The ranking claim assumes that all scores quoted from the public leaderboard were produced under identical, fair evaluation settings for every model; if some sub-10B entries were evaluated under different protocols, being first may be an artifact of protocol differences rather than model quality.

Editorial extensions

If this is right

  • Other groups can reproduce the recipe on a 7B language model—NaViT-style encoder, about two million pretraining captions, filtered instruction data, and model soup—without needing tens of billions of tokens.
  • Native dynamic resolution removes the tile-splitting step, so documents, screenshots, and charts of arbitrary sizes can be served in a single packed forward pass, simplifying deployment.
  • The Chinese data pipeline, built by translating existing sets, generating answers with a strong VLM, and verifying by human labelers, offers a path to bilingual capability when open Chinese instruction data is scarce.
  • The slightly negative effect of removing image-free questions suggests that some text-only samples belong in visual instruction tuning, making data filtering a question of balance rather than simple elimination.
  • A 7B model beating several-times-larger models on the leaderboard would lower the compute bar for real-world multimodal applications.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The abstract says fewer than 4 billion tokens while the training section says less than 5 billion and the table sums to about 4.4 billion; reconciling this figure would sharpen the headline claim.
  • The paper's observation that dropping no-image questions slightly hurt performance is a clue that pure-text data acts as a regularizer, which a controlled study could test by adding a small text-only set to a VLM training run.
  • The leaderboard claim is a snapshot from December 8, 2024, so it is necessarily time-limited; re-checking after later sub-10B models are submitted would confirm whether the ranking is durable.
  • An ablation isolating the NaViT encoder from the data filtering would show how much of the gain comes from resolution handling versus data curation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. POINTS1.5 is a 7B-parameter vision-language model built on the authors' earlier POINTS1.0. The paper replaces a fixed-resolution CLIP encoder with a NaViT-style dynamic-resolution encoder, adds Chinese pre-training and instruction-tuning data, and applies LLM-based filtering to visual instruction datasets. The headline claims are that the model is trained on fewer than 4 billion tokens and ranks first on the OpenCompass leaderboard among sub-10B models. The paper reports architecture details, dataset construction, training configurations in Table 3, and evaluation against open-source and proprietary models in Tables 4-5.

Significance. If the headline results held, the contribution would be practically significant: a competitive sub-10B open VLM with a very small training-token budget, bilingual support, and native arbitrary-resolution image processing. The paper also releases model weights and code links, which supports reproducibility. However, the efficiency claim is internally inconsistent with the paper's own token counts, and the ranking claim is weakened by model selection on the evaluation benchmark itself. These issues need to be resolved before the headline claims can be taken at face value.

major comments (3)
  1. [Abstract; Section 5; Table 3] The abstract states that POINTS1.5-7B is 'trained on fewer than 4 billion tokens,' but Section 5 states the total is 'less than 5 billion tokens,' and Table 3 reports about 2.1B pre-training tokens plus about 2.3B visual instruction tuning tokens, which sums to about 4.4B. Under the paper's own accounting the '<4B' claim is false; the authors should either correct the abstract or explain which tokens are excluded from the count.
  2. [Section 5, 'Model Soup over Best Performing Model'; Section 6, Tables 4-5] The final model is selected by averaging models with the best OpenCompass scores, and the same OpenCompass benchmark suite is then used in Table 4 to support the ranking claim. Because the model soup composition is chosen from these very scores, the reported margin over Qwen2-VL-7B (about 0.3 average points) is not an unbiased estimate. The authors should report robustness, e.g., multiple soup seeds, or evaluation on a held-out benchmark not used in selection.
  3. [Section 4, 'Visual Instruction Tuning Set Filtering'] The abstract claims that the filtering methods were 'comprehensively evaluate[d]' and the best ones selected, but Section 4 provides no ablation table or quantitative comparison. In particular, the image-free-answer filtering is reported to slightly degrade performance, yet the paper does not state whether the degraded variant was used in the final model. Adding ablations or removing the 'comprehensive evaluation' claim is required to support the contribution.
minor comments (3)
  1. [Figure 3 caption] The caption contains a typo: 'converntional' should be 'conventional.'
  2. [Table 5] The table lists two-number entries (e.g., 20.4/18.4 for InternVL2-8B) without defining what the two values represent; a footnote or legend is needed.
  3. [Section 2, batch forwarding] The phrase 'image sequence[Dao, 2024]' should have a space before the citation bracket.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline claims are external-benchmark results, and self-citations to POINTS1.0 are non-load-bearing engineering references; the abstract's token-count discrepancy is a factual inconsistency, not a circular step.

full rationale

POINTS1.5 is an empirical systems report rather than a derivation, so its claims are not generated by a formal chain of equations. The two headline assertions, namely the token budget and the OpenCompass rank, come from training logs and an external leaderboard, not from the model's own definitions. The only mathematical expressions in the paper, Eq. (1) for CapFusion and Eq. (2) for perplexity, are data-curation tools and are never used to derive the performance numbers. Self-citations to POINTS1.0 for dataset construction, the two-stage training recipe, and model soup are references to prior engineering choices; none of them is a uniqueness theorem, a smuggled ansatz, or a fitted parameter that forces the reported result. The model-soup procedure and the filtering-method selection do use OpenCompass scores as the selection criterion, which is benchmark fitting rather than circular reasoning: the reported score is a measured value on an external benchmark, not a quantity that is equal to the selection criterion by construction. A genuine internal inconsistency exists, because the abstract says 'fewer than 4 billion tokens,' while Table 3 lists about 2.1B pre-training tokens plus about 2.3B instruction-tuning tokens, summing to roughly 4.4B, and Section 5 says 'less than 5 billion tokens'; however, this is an arithmetic or reporting error, not a circularity. Under the required standard that circularity be exhibited by a specific reduction or a fitted parameter renamed as a prediction, no circular step can be identified.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. Its free parameters are standard training hyperparameters and data-selection thresholds, and its axioms are common domain assumptions in VLM development. The main load-bearing premise is the validity of the OpenCompass benchmark as a fair comparison tool.

free parameters (7)
  • Perplexity filtering percentile = 20% (approximately 1 million Chinese captions)
    Section 3, Chinese Pre-training Dataset: the authors select the first 20% of captions sorted by perplexity to form the Chinese pre-training subset.
  • Pre-training learning rate = 2e-4
    Table 3 lists learning rate 2e-4 for the pre-training stage.
  • Instruction tuning learning rate = 2e-5
    Table 3 lists learning rate 2e-5 for visual instruction tuning.
  • Batch size = 32
    Table 3 lists batch size 32 for both stages.
  • Context length = 4096
    Table 3 lists context length 4096.
  • Training tokens count = ~2.1B pre-training + ~2.3B instruction tuning
    Table 3 reports these token counts, which sum to 4.4B, conflicting with the abstract's 'fewer than 4 billion'.
  • Model soup composition = not specified
    Section 5 says the final model is a soup of models trained with different instruction tuning datasets and epochs, but does not specify the number of models or selection criteria beyond OpenCompass score.
assumptions (3)
  • domain assumption The LLaVA-style architecture (vision encoder, MLP projector, LLM) is an effective base for vision-language post-training.
    The authors adopt this architecture without formal justification, citing prior works like LLaVA-Next and InternVL.
  • domain assumption The pre-trained Qwen2.5-7B-Instruct LLM and the NaViT-style vision encoder are high-quality starting points.
    Section 2 states the LLM is initialized from Qwen2.5-7B-Instruct and the vision encoder follows Qwen2-VL/Idefics2; no independent evaluation of these components is provided.
  • domain assumption The OpenCompass benchmark suite provides a fair and reliable measure of real-world VLM capability.
    Section 6 uses OpenCompass leaderboard scores as the primary evidence for the headline claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of POINTS1.5: Building a Vision-Language Model towards Real World Applications." pith.science (2026). https://pith.science/paper/URGVIDG4

@misc{pith2026241208443,
  author       = {Pith},
  title        = {Pith review of: POINTS1.5: Building a Vision-Language Model towards Real World Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/URGVIDG4}},
  note         = {Machine review of arXiv:2412.08443}
}
read the original abstract

Vision-language models have made significant strides recently, demonstrating superior performance across a range of tasks, e.g. optical character recognition and complex diagram analysis. Building on this trend, we introduce a new vision-language model, POINTS1.5, designed to excel in various real-world applications. POINTS1.5 is an enhancement of POINTS1.0 and incorporates several key innovations: i) We replace the original CLIP vision encoder, which had a fixed image resolution, with a NaViT-style vision encoder that supports native dynamic high resolution. This allows POINTS1.5 to process images of any resolution without needing to split them into tiles. ii) We add bilingual support to POINTS1.5, significantly enhancing its capability in Chinese. Due to the scarcity of open-source Chinese datasets for vision-language models, we collect numerous images from the Internet and annotate them using a combination of manual and automatic methods. iii) We propose a set of rigorous filtering methods for visual instruction tuning datasets. We comprehensively evaluate all these filtering methods, and choose the most effective ones to obtain the final visual instruction tuning set. Thanks to these innovations, POINTS1.5 significantly outperforms POINTS1.0 and demonstrates strong performance across a range of real-world applications. Notably, POINTS1.5-7B is trained on fewer than 4 billion tokens and ranks first on the OpenCompass leaderboard among models with fewer than 10 billion parameters

Figures

Figures reproduced from arXiv: 2412.08443 by the authors.

Figure 1
Figure 1. Performance of Open-Source Models on the OpenCompass Leaderboard[Contributors, 2023]. POINTS1.5 ranks first among all models under 10B in size, even outperforming models several times larger. The size of each bubble represents the model size. Abstract Vision-language models have made significant strides recently, demonstrating su￾perior performance across a range of tasks, e.g. optical character recognition and comp… view at source ↗
Figure 2
Figure 2. POINTS1.5 shows great potential to solve challenging real world problems. existing vision-language model, such as Qwen2-VL-72B, to extract text from these images. Human labelers then verified these annotations, correcting minor errors or discarding them if the errors were significant. Visual Instruction Tuning Set Filtering. We manually reviewed each dataset used in POINTS1.0 and identified two significant issues: i… view at source ↗
Figure 3
Figure 3. POINTS1.5 uses the converntional LLaVA-style architecture, consisting of a vision encoder, a MLP projector and a LLM [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: The chat template during pre-training in POINTS1.0 (above) and POINTS1.5 (below) Chinese Pre-training Dataset Following POINTS1.0, we employ a two-step procedure to create the pre-training dataset: i) We use CapFusion [Yu et al., 2024] to merge the caption generated by…
Figure 5
Figure 5. Figure 5: Prompts used in the chat template during pre-training stage. Chinese Visual Instruction Tuning Dataset We inherit all visual instruction tuning datasets from POINTS1.0, except for those generated in this section. To create Chinese visual instruction tuning datasets, we…
Figure 6
Figure 6. Figure 6: Prompts to create the Chinese OCR datasets. After the creation of Chinese datasets, we obtain the distribution across 9 categories and the En￾glish&Chinese distribution for the final visual instruction tuning datasets we used in POINTS1.5. We observe a significant imba…
Figure 7
Figure 7. Figure 7: Distribution of visual instruction tuning data in POINTS1.5. The left figure shows the distribution across different categories, and the right figure shows the distribution between English and Chinese. 4 Visual Instruction Tuning Set Filtering Before filtering the visu…
Figure 8
Figure 8. Figure 8: Procedure to filter out samples containing grammatical errors (a) and distribution between grammatically correct samples and samples containing grammatical errors (b). Questions can be answered without images. It is common sense that the data used to train a vision-lan…
Figure 9
Figure 9. Figure 9: Questions can be answered without referring to the image. 5 Training and Model Strategy 0 4 8 12 16 20 24 28 32 Unfrozen Layers 61 62 63 64 65 66 OpenCompass Average Score lr=3e-5 lr=2e-5 lr=1e-5 [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Unfreezing the vision encoder during pre [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 10
Figure 10. Figure 10: Data samples containing grammatical errors (marked with red) in visual instruction tuning set. trained separately (e.g., the Qwen2-VL vision encoder used in POINTS1.5). Subsequently, the vision encoder is fixed, and the projector and LLM are trained end-to-end. In pra…
Figure 12
Figure 12. Figure 12: We envision that extending a large language model with additional modalities using LLaVA-style architecture should follow the three-stage procedure illustrated in this figure. The three icons on the left denote the status of each module during the three stages. From l…
Figure 13
Figure 13. Figure 13: OCR and reasoning ability. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Complex Chinese OCR 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Complex OCR 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Summarize key points from an image. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Latex formula extraction 20 [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Mathematical problem solving 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Image translation 22 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Object identification. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Key information extraction and reasoning. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    POINTS-Seeker-8B is an 8B multimodal model trained from scratch for agentic search that uses seeding and visual-space history folding to outperform prior models on six visual reasoning benchmarks.

  2. InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

    cs.CV 2025-01 conditional novelty 6.0 of 10

    IXC-2.5-Reward is an open-source multimodal reward model that achieves 70.0% macro accuracy on VL-RewardBench and improves LVLM chat via PPO.

  3. Ocean-OCR: Towards General OCR Application via a Vision-Language Model

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A 3B vision-language model trained with a large OCR-heavy data mix reports top scores on OCR benchmarks and beats the TextIn and PaddleOCR engines on custom document, scene-text, and handwriting evaluations.

  4. Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Valley2, a 7B-scale open-source multimodal model, reports second-best OpenCompass average (67.4) among sub-10B models and the highest score (79.66) on its own in-house Ecom-VQA benchmark.

Reference graph

Works this paper leans on

55 extracted references · 6 canonical work pages · cited by 4 Pith papers

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

    arXiv 2024

  3. [3]

    Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks

    Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410.10563, 2024 a

    arXiv

  4. [4]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 b

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 b

    arXiv 2024

  5. [5]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024 c

    arXiv 2024

  6. [6]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  7. [7]

    Nvlm: Open frontier-class multimodal llms

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402, 2024

    arXiv 2024

  8. [8]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

Show all 55 references
  1. [9]

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neura...

    2024

  2. [10]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wa...

    2024 arXiv

  3. [11]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, a...

    2024 arXiv

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  5. [13]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.11691, 2024

    2024 arXiv

  6. [14]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913, 2017

    2017

  7. [15]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  8. [16]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709, 2019

    2019

  9. [17]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. ArXiv, abs/1603.07396, 2016. URL https://api.semanticscholar.org/CorpusID:2682274

    2016 arXiv

  10. [18]

    Visual information extraction in the wild: practical dataset and end-to-end solution

    Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual information extraction in the wild: practical dataset and end-to-end solution. In International Conference on Document Analysis and Recognition, pages 36--53. Springer, 2023

    2023

  11. [19]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Lauren c on, L \'e o Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    2024 arXiv

  12. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  13. [21]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023 a

    2023 arXiv

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR, 2023 b

    2023

  15. [23]

    Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models

    Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models...

    arXiv

  16. [24]

    Improved baselines with visual instruction tuning, 2023 b

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023 b

    2023

  17. [25]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a . URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  19. [27]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023 c

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023 c

    2023 arXiv

  20. [28]

    Rethinking overlooked aspects in vision-language models

    Yuan Liu, Le Tian, Xiao Zhou, and Jie Zhou. Rethinking overlooked aspects in vision-language models. arXiv preprint arXiv:2405.11850, 2024 c

    2024 arXiv

  21. [29]

    Points: Improving your vision-language model with affordable strategies

    Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, and Jie Zhou. Points: Improving your vision-language model with affordable strategies. arXiv preprint arXiv:2409.04828, 2024 d

    2024 arXiv

  22. [30]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023 d

    2023 arXiv

  23. [31]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976--11986, 2022

    2022

  24. [32]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024 a

    2024 arXiv

  25. [33]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 0 2507...

    2022

  26. [34]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  27. [35]

    Ovis: Structural embedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024 b

    2024 arXiv

  28. [36]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204, 2019

    2019

  29. [37]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  30. [38]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. Technical Report 1, 2, 9, 10, OpenAI, 2023. URL https://example.com/gpt4-technical-report

    2023

  31. [39]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Informa...

    2022

  32. [40]

    Neural machine translation of rare words with subword units

    Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  33. [41]

    Fast wordpiece tokenization

    Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Fast wordpiece tokenization. arXiv preprint arXiv:2012.15524, 2020

    2012 arXiv

  34. [42]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  35. [43]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023

    2023 arXiv

  36. [44]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804, 2024 a

    2024 arXiv

  37. [45]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 b

    2024 arXiv

  38. [46]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024 c

    2024 arXiv

  39. [47]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022 arXiv

  40. [48]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  41. [49]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  42. [50]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023

    2023 arXiv

  43. [51]

    Capsfusion: Rethinking image-text data at scale

    Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14022--14032, 2024

    2024

  44. [52]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  45. [53]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  46. [54]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning. arXiv preprint arXiv:2409.20566, 2024

    2024 arXiv

  47. [55]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169--186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169--186. Spri...

    2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.