Pith. sign in

REVIEW 4 major objections 5 minor 64 references

Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that letting a vision-language model encode images at their native resolution and aspect ratio, rather than resizing or cropping, produces large gains on fine-grained visual tasks and can be cleanly tested with a new…

desk verdict Controlled evidence that native-resolution encoding beats cropping for fine-grained VLM tasks, but the central benchmark is unreleased and under-specified, so the claim is solid-but-conditional. read the letter →

arxiv 2506.12776 v1 pith:RSLFPNGH submitted 2025-06-15 cs.CV

Junbo Niu , Yuanhong Zheng , Ziyang Miao , Hejun Dong , Chunjiang Ge , Hao Liang , Ma Lu , Bohan Zeng
show 3 more authors
Qiahao Zheng Conghui He Wentao Zhang
This is my paper
classification cs.CV
keywords vision-languagemodelsnativeresolutionvisualencodingresolution-centricbenchmarkaspectratiorobustnessdynamicinstructiontuningOCRanddocumentunderstandingsequencepacking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models usually resize every image to a fixed low resolution before encoding, which destroys detail in small text, dense charts, and extreme panoramas. The paper claims that this "resolution dilemma" has two parts: most training frameworks cannot natively encode arbitrary resolutions, and most benchmarks do not measure resolution sensitivity. It introduces RC-Bench, a benchmark of 1,750 images deliberately balanced across seven area levels and five aspect-ratio categories, with human-verified question-answer pairs, and NativeRes-LLaVA, an open training framework whose vision encoder works directly at an image's original resolution and shape. Across resolution-centric benchmarks the native strategy substantially outperforms fixed and cropping-based strategies, while leaving semantic-centric benchmarks largely unchanged.

What carries the argument

The load-bearing mechanism is a native-resolution vision encoder: a ViT with 2D rotary position embeddings that accepts arbitrary image sizes and produces a variable number of visual tokens via 2x2 average pooling, a 4x patch merger, and an MLP that projects features into the language model. Because native resolution yields variable-length patch sequences, training uses Patch n' Pack style packing so multiple images' patches are concatenated into one sequence and attention is isolated per image with variable-length flash attention. On the evaluation side, RC-Bench supplies a 7x5 grid of area levels and aspect-ratio categories, and its coefficient-of-variation metrics (ACV and RCV) turn "robustness to resolution variation" into a directly measurable quantity.

What would settle it

Release RC-Bench and re-score every model after downsampling each image to a fixed low resolution while keeping the same questions; if accuracy does not drop sharply for models with high native-resolution scores, the benchmark is not measuring resolution sensitivity and the central claim would be undercut.

Watch

Extended reading notes

Core claim

The central claim is that native-resolution visual encoding, meaning processing each image at its original pixel dimensions and aspect ratio with a variable-token vision transformer instead of resizing or tiling it, is the decisive factor in fine-grained visual understanding. On a matched 1.34M-sample training setup, the proposed model reaches 74.0 on TextVQA, 705 on OCRBench, 89.7 on DocVQA, 79.0 on ChartQA, 61.0 on InfographicVQA, 61.3 on HR-Bench, and 60.1 on RC-Bench, beating cropping-based and hybrid-encoder models that train on much larger data. The advantage is largest on extreme aspect ratios and extreme areas; in the per-cell comparison at matched 728x728 resolution, the native strategy wins or ties in nearly every cell and loses clearly in only one "sweet spot" configuration that matches the pretraining distribution. The paper frames this as resolving a dilemma: native encoding is needed for resolution-centric tasks, while semantic-centric tasks are resolution-insensitive, so benchmarks and models must separate the two.

Load-bearing premise

The load-bearing premise is that RC-Bench's 1,750 images and their human-verified question-answer pairs truly isolate resolution and aspect ratio as the factors that matter; if those images or questions are biased, noisy, or not actually resolution-sensitive, the central comparison loses its evidentiary force.

Editorial extensions

If this is right

  • For fine-grained OCR, document, chart, and infographic tasks, preserving original resolution and aspect ratio should be expected to beat cropping-based pipelines even when the latter see more training data.
  • Benchmark reporting should include accuracy breakdowns by area and aspect ratio, plus coefficients of variation, because average accuracy hides which models fail on extreme inputs.
  • Raising the maximum supported input resolution in a native-resolution model improves resolution-centric scores while leaving semantic-centric scores roughly flat, so high MaxRes is a targeted design choice rather than a universal upgrade.
  • Smaller language models using native resolution can outperform larger ones using cropping: a 2B-scale native model already beats a 7B cropping-based baseline on several resolution-sensitive benchmarks.
  • The open modular training stack makes it possible to swap vision encoders, pack multimodal sequences, and run elastic native-resolution training, which the paper argues the open-source community previously lacked.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RC-Bench's proprietary images were replaced by an independently built public version with the same balancing protocol, the reported native-versus-crop gaps could be checked for benchmark-specific bias; this is the most direct way to test whether the conclusion generalizes beyond the paper's selected 1,750 images.
  • The single cell where cropping wins suggests a concrete hybrid design: route images whose size and shape match the pretraining distribution through a fixed-resolution path, and send every other image through the native path; this could be tested while holding all other training conditions fixed.
  • Because the vision encoder is initialized from a strong pretrained native-resolution model rather than trained from scratch, part of the gain may reflect the pretrained encoder's quality; a from-scratch native ViT trained at scale is the natural test of the framework's own contribution.
  • A resolution-aware benchmark like RC-Bench could also serve as a diagnostic for deployed models: plotting accuracy as a function of downsampling factor would reveal at what resolution each model's fine-grained understanding breaks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses what it calls the 'Resolution Dilemma' in vision-language models (VLMs): most models rely on fixed low-resolution visual encoders, and existing benchmarks do not systematically vary image resolution and aspect ratio. The authors introduce two artifacts: RC-Bench, a new benchmark with a balanced distribution over seven area levels and five aspect-ratio categories, and NativeRes-LLaVA, an open-source training framework that encodes images at native resolution using a 2D-RoPE ViT with sequence packing. Experiments compare native-resolution encoding against cropping, hybrid, and fixed-resolution strategies, reporting that native encoding substantially improves performance on 'Resolution-Centric' benchmarks (e.g., TextVQA, DocVQA, OCRBench, RC-Bench) while roughly preserving performance on 'Semantic-Centric' benchmarks. A controlled ablation in Table 5 swaps the same Qwen2-VL ViT into both the cropping and native pipelines, and the native strategy still wins on most resolution-centric tasks. The paper includes code and detailed training hyperparameters, and it candidly discusses limitations in Section 6.

Significance. If the claims hold, the paper makes two useful contributions. First, NativeRes-LLaVA is a genuinely open and modular framework for native-resolution visual encoding, which fills a gap in the open-source ecosystem relative to closed models like Qwen2-VL and Kimi-VL; the code release and detailed training recipes are commendable. Second, the controlled comparison in Table 5—using the same Qwen2-VL ViT for both the cropping and native strategies—is a strong experimental design that isolates the visual-encoding strategy from the backbone choice, and it provides credible evidence that native-resolution encoding yields real gains. The paper also honestly discusses training-data mismatches and computational bottlenecks. However, the central empirical evidence on RC-Bench is weakened by the benchmark being unreleased and by the undisclosed details of its construction and evaluation pipeline; moreover, the Semantic-Centric versus Resolution-Centric taxonomy is derived from an undescribed manual sensitivity analysis. These issues currently prevent the benchmark-based claims from being fully audited.

major comments (4)
  1. [Section 3.3 and Appendix A.4] RC-Bench's validity as a measuring instrument is not adequately established. The benchmark is not released, its construction uses undisclosed 'proprietary data resources,' and the paper does not report the distribution of answer types (numbers, dates, phrases, addresses, sentences, etc.) across the 35 area/ratio cells. Because short answers are scored with Exact Match and long answers with ANLS, and because Appendix A.4 applies additional human-defined normalization rules, any correlation between answer type and area or ratio would directly contaminate the per-cell accuracies and the ACV/RCV statistics in Tables 6-8. The authors should release the benchmark or, at minimum, provide the full per-cell answer-type breakdown, per-cell image counts, inter-annotator agreement statistics for the human verification step, and a description of how the answer-type classification was validated.
  2. [Section 3.1] The manual sensitivity analysis that classifies benchmarks into Semantic-Centric and Resolution-Centric types is not described. The paper states that 'we systematically adjusted the components of VLM and resolution processing strategies' to analyze sensitivity, but it does not report the exact procedure: which model variants, which resolution ranges, which thresholds defined 'sensitivity,' and how manual case analysis entered the classification. This taxonomy is load-bearing because it selects the evaluation benchmarks and supports the claim that native resolution helps only Resolution-Centric tasks (Tables 3-5 and Figure 3). Without a reproducible procedure, the reader cannot verify that the benchmark categorization is objective rather than shaped by the authors' expectations.
  3. [Section 5.4.1 and Table 5] The controlled comparison between Cropping and Native on the same Qwen2-VL ViT is not fully specified. The text says the ViT in LLaVA-NeXT was replaced with Qwen2-VL's ViT and its input resolution fixed to match SigLip-384, but it does not explain how the cropping/tiling mechanism was adapted to the new ViT: specifically, how the global thumbnail and local tiles are encoded, whether the 4x patch-merging compression module is applied to both pipelines, and how positional embeddings are handled for non-native fixed resolutions. Without this detail, the observed gains of NativeRes-LLaVA over LLaVA-NeXT-QwenViT could partly reflect an improperly tuned cropping pipeline rather than the native-resolution strategy itself. Please provide precise architecture specifications and token-budget equivalence for both compared pipelines.
  4. [Section 5.4.2 and Table 8] The headline comparison of 47.9 (Crop) versus 53.6 (Native) on RC-Bench rests entirely on the benchmark's soundness, which is currently not verifiable (see the first major comment). Even setting that aside, the paper does not report per-cell sample sizes or confidence intervals. With 1,750 images spread over 35 cells, the average cell has only 50 images, and some cells could be much smaller; the reported accuracy differences (e.g., -0.29 at (NM,B) and +0.24 at (AH,A)) may not be statistically reliable. The authors should provide per-cell counts and bootstrapped confidence intervals for the accuracy differences in Figure 6(c) and for the ACV/RCV values in Tables 6-8.
minor comments (5)
  1. [Section 5.2] The evaluation protocol list has a duplicated numbering: item (5) appears twice, once for 'SC-type tasks' and once for 'RC-Bench.' This should be renumbered.
  2. [Tables 6-8] The ACV and RCV values are described as 'multiplied by 10^2' only in the table captions; the main text in Section 5.4.2 should also state this scaling to avoid misinterpretation of the metric magnitudes.
  3. [Section 3.3 and Appendix A.1] The paper says RC-Bench images are 'uniformly distributed across seven predefined resolution levels and five aspect ratio categories,' but Figure 9(a) shows unequal color intensities across cells. Please clarify whether the target distribution is exactly uniform or approximately balanced, and report the actual counts per cell.
  4. [Reference [38]] The NaViT reference is malformed: the author list reads 'Basil Mustafa Mostafa Dehghani, Jonathan Heek Josip Djolonga, et al.' and the title is incorrectly rendered. Please fix the citation to include the full author list and correct title.
  5. [Section 5.4.2] The heatmap color legend in Figure 6 is incomplete: the text mentions green, red, and yellow cells, but the figure does not show a colorbar or threshold values. A clear legend would help readers interpret the magnitude of differences.

Circularity Check

1 steps flagged · score 3.0 of 10

Mild self-referentiality: the SC/RC taxonomy is defined and then 'validated' with the authors' own model, and headline results use the authors' own unreleased RC-Bench; however, controlled ablations on external benchmarks independently support the central conclusion, so no prediction reduces by construction to a fitted input or to a self-citation chain.

  1. self definitional [Section 3.1 (benchmark categorization) validated in Section 5.4.1 (Table 5 analysis).]
    "We systematically adjusted the components of VLM [28] and resolution processing strategies to analyze the sensitivity of the selected benchmark to the input image resolution. We categorized these benchmarks according to their sensitivity to visual encoding strategies into two main types: Semantic-Centric (SC) type and Resolution-Centric (RC) type, as presented in Table 2. ... These results validate the resolution dilemmas discussed in Section 3.1 and support the effectiveness of our proposed classification of multimodal benchmarks into Semantic-Centric (SC) and Resolution-Centric (RC) tasks."

    The RC/SC labels are assigned by an empirical sensitivity criterion: a benchmark is called Resolution-Centric when VLM performance changes measurably with input resolution, and that measurement is performed with the authors' own model, NativeRes-LLaVA (Section 3.1, Figure 3). Section 5.4.1 then 'validates' the taxonomy by observing that performance on the RC-type benchmarks changes with input resolution, again using the same model. The validating observation is largely entailed by the defining criterion, so this confirmation loop adds little independent evidence that these benchmarks are resolution-sensitive; the loop is partly tautological rather than an external check.

full rationale

The claimed derivation chain is: (i) benchmark analysis shows existing benchmarks under-represent resolution and aspect-ratio diversity; (ii) a new resolution-centric benchmark (RC-Bench) is constructed; (iii) a native-resolution encoding framework (NativeRes-LLaVA) is trained on standard public SFT data; (iv) controlled ablations show native encoding outperforms cropping on RC-Bench and external RC-type benchmarks. Step (iii) fits no parameter to any evaluation set: training uses only LLaVA-Pretrain, LLaVA-mix665k, and LLaVA-NeXT-Data, and RC-Bench scores are evaluations, not training targets. Step (iv) is the crux: the Native-versus-Crop comparison at equal MaxRes with an identical Qwen2-VL-ViT backbone (Tables 5 and 8) favors native encoding on external, pre-existing benchmarks (TextVQA 72.7 vs 70.2, OCRBench 685 vs 666, DocVQA 80.2 vs 77.5, ChartQA 78.7 vs 76.2 at 728x728 with 1.34M data) as well as on RC-Bench, so the central claim does not reduce to the self-built instrument. The benchmark is not stacked in favor of the method: the paper reports one (NM, B) cell where cropping beats native resolution by 0.29 and explains it via pre-training alignment, which indicates a non-rigged instrument. Two genuine but mild self-referential elements remain. First, the SC/RC taxonomy is derived from resolution-sensitivity measurements made with the authors' own NativeRes-LLaVA and then 'validated' with the same model; that validation is largely entailed by the category definition and is flagged as a partial self-definitional step. Second, headline RC-Bench numbers (Tables 6-8, Figure 6) are produced by the authors on their own unreleased benchmark, whose answer-type-dependent EM/ANLS protocol (Appendix A.4), undisclosed 'proprietary data resources' (Section 3.3), and lack of inter-annotator statistics are not independently auditable; this is a benchmark-validity and reproducibility risk rather than a reduction of the result to its inputs, because the QA pairs were generated by GPT-4o with human verification, not by the proposed method. Self-citations ([39] Ovo-Bench and [57] InternLM-XComposer2.5-OmniLive, both co-authored by Junbo Niu) are contextual and not load-bearing. The paper's own Limitations section concedes non-SOTA status, inherited ViT initialization, and inadequate high-resolution training data, further indicating that the claims are empirical rather than definitional.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim depends on the quality of the benchmarks used for evaluation, especially the newly introduced RC-Bench, which is not released and relies on undisclosed proprietary data and GPT-4o-generated QAs. The comparison also depends on hand-chosen token caps and resolution limits, and on the unverifiable SC/RC categorization of existing benchmarks. These are not fitted parameters in the sense of a numerical fit, but they are design choices that affect the measured improvements and should be justified and released for full audit.

free parameters (3)
  • Max tokens per image = 2048 (SFT1), 4096 (SFT2)
    Chosen by hand to fit GPU memory and capped per stage; different token caps across configurations confound comparisons between methods. Location: Section 5.1 and Appendix A.2.
  • Maximum resolution (MaxRes) = 378x378, 728x728, 1260x1260, 1792x1792
    Hand-selected resolution limits used to sweep the effect of resolution; the paper does not justify the specific values beyond experimental coverage. Location: Tables 5, 6, and 9.
  • ANLS threshold = not stated
    ANLS is used for long answers, but the similarity threshold (commonly 0.5) is not specified, leaving the exact match criterion partially undefined. Location: Section 3.3 Evaluation methodology.
assumptions (4)
  • domain assumption The Qwen2-VL native-resolution ViT, with its pretrained weights, is a suitable and unbiased vision encoder for comparing resolution strategies.
    The paper initializes its vision encoder from Qwen2-VL-2B and uses it for both native and cropping ablations, assuming no hidden incompatibility or advantage for one strategy. Location: Section 5.1 and Table 5.
  • domain assumption GPT-4o-generated QA pairs, after human review, are visually grounded and correct to the degree needed for accurate accuracy measurement.
    RC-Bench QA quality is not independently verified; no inter-annotator agreement or quality audit is reported. Location: Section 3.3 Generation of question-answer pairs.
  • domain assumption NaViT Patch n' Pack and variable-length FlashAttention correctly isolate sub-sequences and introduce no artifacts when packing images of different resolutions.
    The framework relies on these external mechanisms as described in cited work, without independent verification in this paper. Location: Section 4.3.
  • ad hoc to paper The manual sensitivity analysis that sorts benchmarks into Semantic-Centric and Resolution-Centric is valid, though its procedure is not described.
    The categorization drives the interpretation of results, but the experiment is summarized in a single sentence with no details on models, resolutions, or metrics. Location: Section 3.1, paragraph above Table 2.
invented entities (2)
  • RC-Bench
    purpose: A new benchmark to measure VLM accuracy across resolution and aspect ratio bins.
    The benchmark is not publicly released; the paper provides only a code link for the framework, so its questions and labels cannot be independently checked. Location: Section 3.3 and Abstract.
  • Semantic-Centric vs Resolution-Centric task taxonomy
    purpose: A conceptual split used to explain when native resolution matters and to select evaluation benchmarks.
    The taxonomy is derived from the authors' own sensitivity experiments and manual case analysis, without external validation or a reproducible protocol. Location: Section 3.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models." pith.science (2026). https://pith.science/paper/RSLFPNGH

@misc{pith2026250612776,
  author       = {Pith},
  title        = {Pith review of: Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSLFPNGH}},
  note         = {Machine review of arXiv:2506.12776}
}
read the original abstract

Vision-Language Models (VLMs) face significant challenges when dealing with the diverse resolutions and aspect ratios of real-world images, as most existing models rely on fixed, low-resolution inputs. While recent studies have explored integrating native resolution visual encoding to improve model performance, such efforts remain fragmented and lack a systematic framework within the open-source community. Moreover, existing benchmarks fall short in evaluating VLMs under varied visual conditions, often neglecting resolution as a critical factor. To address the "Resolution Dilemma" stemming from both model design and benchmark limitations, we introduce RC-Bench, a novel benchmark specifically designed to systematically evaluate VLM capabilities under extreme visual conditions, with an emphasis on resolution and aspect ratio variations. In conjunction, we propose NativeRes-LLaVA, an open-source training framework that empowers VLMs to effectively process images at their native resolutions and aspect ratios. Based on RC-Bench and NativeRes-LLaVA, we conduct comprehensive experiments on existing visual encoding strategies. The results show that Native Resolution Visual Encoding significantly improves the performance of VLMs on RC-Bench as well as other resolution-centric benchmarks. Code is available at https://github.com/Niujunbo2002/NativeRes-LLaVA.

Figures

Figures reproduced from arXiv: 2506.12776 by the authors.

Figure 1
Figure 1. Left: The images distribution by Area(pixels). Right: The images distribution by Ratio. distribution. Similarly, the distribution of their image aspect ratios is also relatively concentrated. Although these benchmarks provide high-quality and challenging question-answer (QA) pairs, they are insufficient in simulating the inherent diversity of real-world visual data, especially in images covering extreme resolutions … view at source ↗
Figure 2
Figure 2. An illustration of the distinc￾tion between Resolution-Centric (RC) and Semantic-Centric (SC) visual tasks. RC tasks, such as reading the numbers on the left, necessitate high-resolution input. In contrast, SC tasks, such as recognizing the dog on the right, are less dependent on visual clarity and can be performed even with lower-resolution images. To illustrate this problem, let’s borrow a real-life analogy: when … view at source ↗
Figure 4
Figure 4. Above: Generation pipeline of RC-Bench. The details will be discussed in Section3.3. Below: Examples of three QAs in RC-Bench. More samples are shown in the Appendix A.5. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: As the resolution increases, the per￾formance improvement of NativeRes-LLaVA on the Resolution-Centric Benchmark is more pronounced compared to that on the Semantic￾Centric Benchmarks. How to truly achieve human-level dynamic vi￾sual perception? Nowadays, many methods …
Figure 5
Figure 5. Figure 5: The architecture of NativeRes-LLaVA. 4.3 Multimodal sequence packing Traditional ViT (such as CLIP [40] and SigLip [56]) usually process images of arbitrary resolution and aspect ratio into fixed-length patch sequences to facilitate GPU parallel computing. However, the…
Figure 6
Figure 6. Figure 6: Comparison of Visual Encoding Strategy on the RC-Bench. The figure illustrates the change in accuracy of the Native Resolution method (b) compared to the Cropping-based method (a) at a 728 × 728 resolution. (c), Accuracy Difference, shows the value in each cell calcula…
Figure 7
Figure 7. Figure 7: An illustration of the mechanism between Cropping-based [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the image resolution and aspect ratio distribution used during the pretraining [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Data distribution of existing Benchmarks [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Evaluation results of vision-language models on RC-Bench. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Detail evaluation results of LLaVA-NeXT and NativeRes-LLaVA on RC-Bench. Training data sizes: Left — 1.22M, Right — 1.34M. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 12 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, C. V . Jawahar, and Dimosthenis Karatzas. Scene text visual question answering, 2019

    work page 2019

  4. [4]

    Ocean-ocr: Towards general ocr application via a vision-language model

    Song Chen, Xinyu Guo, Yadong Li, Tao Zhang, Mingan Lin, Dongdong Kuang, Youwei Zhang, Lingfeng Ming, Fengyu Zhang, Yuran Wang, et al. Ocean-ocr: Towards general ocr application via a vision-language model. arXiv preprint arXiv:2501.15558, 2025

    arXiv 2025

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  6. [6]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    2024

  7. [7]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    2023

  8. [8]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

Show all 64 references
  1. [9]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lan...

    2019

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  3. [11]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020

    2020

  4. [12]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  5. [13]

    Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning

    Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321, 2024

    2024 arXiv

  6. [14]

    Seed1.5-vl technical report, 2025

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, et al. Seed1.5-vl technical report, 2025

    2025

  7. [15]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pages 390–406. Springer, 2024

    2024

  8. [16]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. S...

    2016

  9. [17]

    Bert: a review of applications in natural language processing and understanding

    Mikhail V Koroteev. Bert: a review of applications in natural language processing and understanding. arXiv preprint arXiv:2103.11943, 2021

    2021 arXiv

  10. [18]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 14

    2024 arXiv

  11. [19]

    Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790, 2024

    2024 arXiv

  12. [20]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  13. [21]

    Tokenpacker: Efficient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024

    2024 arXiv

  14. [22]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  15. [23]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  16. [24]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recogni...

    2024

  17. [25]

    Eagle 2: Building post-training data strategies from scratch for frontier vision-language models

    Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-language models. arXiv preprint arXiv:2501.14818, 2025

    2025 arXiv

  18. [26]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  19. [27]

    LLaV A-NeXT: Improved reasoning, ocr, and world knowledge.https://llava-vl.github.io/blog/ 2024-01-30-llava-next/

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Improved reasoning, ocr, and world knowledge.https://llava-vl.github.io/blog/ 2024-01-30-llava-next/ . 2024

    2024

  20. [28]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  21. [29]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  22. [30]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024

    2024

  23. [31]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  24. [32]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  25. [33]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  26. [34]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024

    2024 arXiv

  27. [35]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  28. [36]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  29. [37]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 15

    2021

  30. [38]

    Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution, 2023

    Basil Mustafa Mostafa Dehghani, Jonathan Heek Josip Djolonga, et al. Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution, 2023

    2023

  31. [39]

    Ovo-bench: How far is your video-llms from real-world online video understanding? In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18902–18913, 2025

    Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-llms from real-world online video understanding? In Proceedings of the Computer Vision and Pattern Recognition ...

    2025

  32. [40]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  33. [41]

    When do we not need larger vision models? In European Conference on Computer Vision, pages 444–462

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In European Conference on Computer Vision, pages 444–462. Springer, 2024

    2024

  34. [42]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  35. [43]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  36. [44]

    Internlm: A multilingual language model with progressively enhanced capabilities, 2023

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023

    2023

  37. [45]

    Kimi-vl technical report

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025

    2025 arXiv

  38. [46]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processi...

    2024

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  40. [48]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  41. [49]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial I...

    2025

  42. [50]

    Parameter-inverted image pyramid networks for visual perception and multimodal understanding

    Zhaokai Wang, Xizhou Zhu, Xue Yang, Gen Luo, Hao Li, Changyao Tian, Wenhan Dou, Junqi Ge, Lewei Lu, Yu Qiao, et al. Parameter-inverted image pyramid networks for visual perception and multimodal understanding. arXiv preprint arXiv:2501.07783, 2025

    2025 arXiv

  43. [51]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  44. [52]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  45. [53]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023

    arXiv

  46. [54]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  47. [55]

    Texthawk: Exploring efficient fine-grained perception of multimodal large language models

    Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng. Texthawk: Exploring efficient fine-grained perception of multimodal large language models. arXiv preprint arXiv:2404.09204, 2024. 16

    2024 arXiv

  48. [56]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023

    2023

  49. [57]

    Internlm-xcomposer2

    Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internlm-xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint arXiv:2412.09596, 2024

    2024 arXiv

  50. [58]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024

    2024 arXiv

  51. [59]

    Llava-uhd v2: an mllm integrating high-resolution feature pyramid via hierarchical window transformer

    Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, et al. Llava-uhd v2: an mllm integrating high-resolution feature pyramid via hierarchical window transformer. arXiv preprint arXiv:2412.13871, 2024

    2024 arXiv

  52. [60]

    Mg-llava: Towards multi-granularity visual instruction tuning

    Xiangyu Zhao, Xiangtai Li, Haodong Duan, Haian Huang, Yining Li, Kai Chen, and Hua Yang. Mg-llava: Towards multi-granularity visual instruction tuning. arXiv preprint arXiv:2406.17770, 2024

    2024 arXiv

  53. [61]

    Swift: a scalable lightweight infrastructure for fine-tuning

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, et al. Swift: a scalable lightweight infrastructure for fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 29733–29735, 2025

    2025

  54. [62]

    Llamafactory: Unified efficient fine-tuning of 100+ language models.arXiv preprint arXiv:2403.13372, 2024

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models.arXiv preprint arXiv:2403.13372, 2024

    2024 arXiv

  55. [63]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 17 A Technical App...

    2025 arXiv

  56. [64]

    $” or measurement units such as “cm

    with our models trained under limited data(Figure 11). The heatmap is structured as a 7 × 5 grid, with Area on the X-axis and Ratio on the Y-axis, visually presenting the model scores across resolution and aspect ratio dimensions. This enables a more intuitive assessment of th...

    2006

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.