Pith. sign in

REVIEW 3 major objections 7 minor 29 references

FILA: Fine-Grained Vision Language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A vision-language model that keeps global context while encoding high-resolution crops outperforms prior models on nine of ten benchmarks.

desk verdict Real architectural contribution, but the headline SOTA numbers are not yet controlled for training data; the abstract's 9.6% doesn't match Table 1's 4.9 points. read the letter →

arxiv 2412.08378 v3 pith:4R6EEAZV submitted 2024-12-11 cs.CV cs.AI

Shiding Zhu , Wenhui Dong , Jun Song , Yingbo Wang , Yanan Guo , Bo Zheng This is my paper · ORCID
classification cs.CVcs.AI
keywords fine-grainedvision-languagemodelshigh-resolutionimageunderstandingdynamiccroppingfragmentationhybridvisualencoderdeepfeaturefusionmultimodallargelanguagemodeldocumentquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fix a specific failure mode in high-resolution vision-language models: dynamic cropping, which cuts an image into sub-images to fit a low-resolution pre-trained encoder, often slices through objects or text and breaks the spatial context. To address this, it introduces HyViLM (also titled FILA), whose Hybrid Encoder pairs a CLIP-ViT processing the global view and crops with a ConvNeXt branch processing the whole image at high resolution; a deep fusion module, CVFM, injects the ConvNeXt features into the ViT's hidden states at four stages. The paper reports that this design outperforms existing MLLMs on nine of ten vision-language benchmarks, with a 9.6% improvement on TextVQA and 6.9% on DocVQA over LLaVA-NeXT. The paper argues that the architecture itself, not extra data or more tokens, is what restores fine-grained recognition.

What carries the argument

The load-bearing mechanism is the Hybrid Encoder with its ConvNeXt-ViT Deep Fusion Module (CVFM). The encoder sends the original image to two branches: CLIP-ViT receives the low-resolution global view plus the dynamically cropped sub-images, while ConvNeXt receives the same image resized to $\frac{32}{14}$ times the crop resolution, preserving aspect ratio. The CVFM aligns each ConvNeXt stage's feature map to the spatial size of the corresponding ViT hidden state, concatenates them along the channel dimension, and runs a gated MLP, $F'_{vl} = F_{vl} + \tanh(\alpha_{\text{dense}}) \cdot \operatorname{MLP}(F_{vl} \oplus F'_{vh})$, so the fusion starts at zero and is learned gradually. Because ConvNeXt's receptive field covers the whole image, each ViT sub-image gains global context, which is the paper's answer to the fragmentation problem.

What would settle it

A decisive test is to train LLaVA-NeXT and HyViLM on identical data and compare all ten benchmarks; if the 9.6% TextVQA gap shrinks or disappears, the advantage is data-driven rather than architectural.

Watch

Extended reading notes

Core claim

The central claim is that image fragmentation from dynamic cropping can be resolved inside the visual encoder rather than patched after the fact. The paper's model keeps the standard two-branch setup of a low-resolution CLIP-ViT and a high-resolution ConvNeXt, but changes where and how they interact: instead of a single cross-attention at the last layer, CVFM concatenates the channel-aligned ConvNeXt features into the ViT's hidden states at four selected layers, with a tanh-gated MLP that lets the fusion be switched on gradually. The paper shows this beats the dynamic-slicing baseline and last-layer interaction baselines on TextVQA, InfoVQA, and MME, and argues that the global context carried by the high-resolution branch is what lets the model read text and objects that fall on crop boundaries.

Load-bearing premise

The entire performance claim rests on the assumption that the reported gains come from the hybrid encoder rather than from the training data, because the main table compares against published baselines with heterogeneous data and only the qualitative section shows a same-data head-to-head.

Editorial extensions

If this is right

  • If the gains are real, any MLLM built on a low-resolution CLIP-ViT can be upgraded to high-resolution document understanding by adding a ConvNeXt branch and the CVFM interaction, without retraining the ViT from scratch.
  • Because the interaction is at four internal layers rather than the last layer, features that would otherwise be lost by the time the final output is computed are preserved; the ablation table supports four layers as the sweet spot.
  • The token count fed to the LLM stays the same as LLaVA-NeXT, so the reported accuracy gains on TextVQA and DocVQA come without extra language-model compute.
  • The two-stage training recipe (low-resolution alignment first, then full high-resolution instruction tuning) is a reusable schedule for other hybrid encoders.
  • The paper's framing suggests OCR and fine-grained VQA are the most direct beneficiaries, since those tasks stress exact reading across crop boundaries; general benchmarks like MME and MathVista also improve, but less dramatically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ablation table suggests that interaction depth, not the fusion operator, is the main driver: channel concatenation beat both local and global cross-attention. An unstated implication is that other dual-encoder MLLMs could gain more by deepening their fusion than by improving the attention scheme.
  • The paper's same-data comparison with LLaVA-NeXT appears only in the qualitative examples; a controlled head-to-head with identical training data across all ten benchmarks would isolate the architectural contribution from the data contribution. That experiment is not in the paper.
  • The described mechanism predicts that performance should degrade gracefully as the high-resolution branch's input resolution is lowered, and that the sweet spot for interaction layers depends on the ViT size. Both are testable without new data.
  • The idea of injecting global context at multiple depths generalizes beyond image cropping: the same CVFM pattern could apply to video frames or multi-page documents, where each frame or page is a 'crop' of a larger scene, though the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes FILA/HyViLM, a high-resolution multimodal large language model that combines a CLIP-ViT encoder with a ConvNeXt auxiliary branch through a new interaction module (CVFM). The Hybrid Encoder interleaves multi-level ConvNeXt features with CLIP-ViT hidden states so that each dynamically cropped sub-image is encoded with access to global context, addressing the semantic fragmentation caused by resolution-adaptive cropping. The model is trained in two stages and evaluated on ten benchmarks, with the paper claiming state-of-the-art results on nine of them, including large reported gains on TextVQA and DocVQA. The main technical novelty is the staged deep fusion between the two visual encoders, and the internal ablations show consistent improvements over last-layer interaction and other fusion variants.

Significance. If the reported gains are caused by the architecture, this is a valuable contribution: dynamic cropping is widely used in high-resolution MLLMs, and the paper identifies a real failure mode (fragmentation at crop boundaries) with concrete qualitative examples. The design of CVFM is clean, the gating initialization at zero is a sensible stability choice, and the internal ablations in Tables 2-4 consistently favor the Hybrid Encoder over the examined alternatives. However, the central 'same setting' claim is not demonstrated: the main table compares against published baselines with heterogeneous training data, the architectural choices are tuned on benchmarks that later appear in the main results, and the headline percentages are not stated in consistent units. The paper is therefore a promising architecture study whose headline empirical claims need stronger controls.

major comments (3)
  1. [§4.3, Table 1; abstract] The abstract and Section 4.3 claim that HyViLM outperforms state-of-the-art MLLMs 'under the same setting,' but the main evidence, Table 1, compares against published numbers from models trained on different data mixtures and with different LLMs and resolutions. The only explicitly same-data comparison is the qualitative Figure 4, which reports no scores. Section 4.1 shows that FILA's instruction mixture includes 28K DocVQA/ChartQA/DVQA/AI2D pairs, and MLLM benchmark numbers are known to be sensitive to such instruction data, so the Table 1 margins could be data-driven rather than architectural. The sentence claiming that MiniGemini-HD's training data are 'almost identical' to FILA's is not backed by a data recipe for MiniGemini-HD. To support the central claim that the Hybrid Encoder and CVFM cause the reported nine-of-ten wins, the paper needs a same-data external baseline (e.g., LLaVA-NeXT or MiniGemini-HD trained on FILA's exact mixture) or a detailed per-dataset comparison of training mixtures.
  2. [§4.4, Tables 2–4] The design choices that define the method—number of ViT interaction layers (4), channel-concatenation fusion, and interpolation resizing—are selected by ablations run on TextVQA, InfoVQA, and MME, and the same three benchmarks appear in the main result table, Table 1. This means the headline architecture is fitted to the evaluation benchmarks, so the Table 1 results do not provide an unbiased estimate of the architecture's advantage. The paper should select hyperparameters on a held-out split, or report results for all ablated configurations on all ten benchmarks, and should include variance over at least several seeds; the reported advantages are often small (e.g., MME 1618/388 vs. 1604/368 and MMMU 41.8 vs. 41.7), and without error bars it is difficult to tell whether they are meaningful.
  3. [Abstract, §1, §4.3] The headline numbers are arithmetically inconsistent in units. Under the natural column alignment of Table 1, HyViLM and LLaVA-NeXT differ by 9.6 points on TextVQA (74.6 vs. 65.0) and by 6.9 points on DocVQA (85.1 vs. 78.2), so the abstract's '9.6% improvement' and '6.9% enhancement' are percentage-point differences, not percentage improvements. Section 4.3, in contrast, reports relative gains: for example, 74.6/71.6 − 1 ≈ 4.2% on TextVQA relative to MiniGemini-HD, and similar relative figures for the other document tasks. The paper should state the reference model and whether differences are absolute points or relative percentages; as written, the central numerical claims cannot be checked against Table 1.
minor comments (7)
  1. [Throughout] The model name is inconsistent: the title and Section 1 introduce FILA, while the abstract, Table 1, Section 4, and Section 5 use HyViLM. Please unify the name throughout.
  2. [§4.3] The claim that HyViLM 'surpasses LLaVA-NeXT by 1.7% on MME' is not directly derivable from Table 1, which reports 1618/388 for HyViLM and 1604/368 for LLaVA-NeXT; please specify which MME component or combined metric is being compared.
  3. [§5 and Table 4] The module name is written as 'CVMF' in Section 5 and in Table 4's note; it should be 'CVFM' to match the rest of the paper.
  4. [Table 5] The alignment table says the example uses a 372 × 372 image, while Section 3.2 and the surrounding text say 336 × 336; please correct the inconsistency.
  5. [§1] The phrase 'our contributions are three folds' should read 'threefold' or 'three contributions.'
  6. [Table 1] The column header 'Doc GeneralTextVQA DocVQA AI2D InfoVQA OCRBenchMMBench MME MMMU MathVista HallBench' is malformed; please list the benchmark names clearly (TextVQA, DocVQA, AI2D, InfoVQA, OCRBench, MMBench, MME, MMMU, MathVista, HallBench).
  7. [§4.4] There is a typo in Table 2: 'Channle Concat' should be 'Channel Concat.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper proposes an architecture and evaluates it empirically; no prediction reduces by construction to a fitted input or to a self-citation.

full rationale

The paper contains no derivation chain of the kind that could be circular: it introduces a Hybrid Encoder and a CVFM fusion module and validates them experimentally. The dynamic-cropping equations (Eqs. 1-4) are definitions of a resolution-selection heuristic, not quantities fitted to the evaluation benchmarks. The fusion formula (Eq. 5) is a designed residual interaction with a zero-initialized gate; it is not derived from or equivalent to any benchmark value. The ablations in Tables 2-4 do select hyperparameters (number of interaction layers, fusion mode, resize method) using TextVQA, InfoVQA, and MME, and those same benchmarks appear in the main results; this is a legitimate benchmark-selection and potential overfitting concern, but it is not circularity because the paper never presents those numbers as independent predictions derived from first principles, and the headline SOTA comparisons are against external published baselines rather than quantities reconstructed from the same fitted choices. The only identifiable self-citation (Zheng et al. 2023 in Related Work) is incidental context and is not load-bearing for any claim. Accordingly, no step in the paper reduces by construction to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities. The free parameters are design choices tuned on the evaluation benchmarks, and the domain assumptions concern the transferability of pretrained encoders and the comparability of baselines. The most load-bearing assumption is that the reported gains are architectural rather than data-driven.

free parameters (4)
  • Number of ViT interaction layers = 4
    Selected via ablation on TextVQA/InfoVQA/MME; 2 or 6 layers perform worse (Table 3).
  • Fusion mode = channel concatenation
    Selected via ablation; channel concat outperforms local CA, global CA, and addition (Table 4).
  • Resizing method for ConvNeXt features = interpolation
    Selected via ablation; interpolation slightly outperforms convolution (Table 4).
  • High-resolution scale factor = 32/14 x Hh
    Chosen so ConvNeXt output dimensions align with concatenated ViT hidden states; no ablation justifying this exact factor.
assumptions (3)
  • domain assumption Pretrained CLIP-ViT and ConvNeXt-L encoders provide useful visual features when combined
    The whole method relies on the transferability of these encoders (Section 3.2, Appendix B).
  • domain assumption Image fragmentation caused by dynamic cropping is a major source of MLLM error on high-resolution tasks
    Motivates the Hybrid Encoder; supported only by qualitative examples in Figure 1, not a systematic analysis (Section 1).
  • domain assumption The benchmark results are comparable across models despite different training data and settings
    The main comparison table mixes published numbers and OpenCompass metrics; the paper claims 'same setting' only in the abstract (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FILA: Fine-Grained Vision Language Models." pith.science (2026). https://pith.science/paper/4R6EEAZV

@misc{pith2026241208378,
  author       = {Pith},
  title        = {Pith review of: FILA: Fine-Grained Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4R6EEAZV}},
  note         = {Machine review of arXiv:2412.08378}
}
read the original abstract

Recently, there has been growing interest in the capability of multimodal large language models (MLLMs) to process high-resolution images. A common approach currently involves dynamically cropping the original high-resolution image into smaller sub-images, which are then fed into a vision encoder that was pre-trained on lower-resolution images. However, this cropping approach often truncates objects and connected areas in the original image, causing semantic breaks. To address this limitation, we introduce HyViLM, designed to process images of any resolution while retaining the overall context during encoding. Specifically, we: (i) Design a new visual encoder called Hybrid Encoder that not only encodes individual sub-images but also interacts with detailed global visual features, significantly improving the model's ability to encode high-resolution images. (ii) Propose an optimal feature fusion strategy for the dynamic cropping approach, effectively leveraging information from different layers of the vision encoder. Compared with the state-of-the-art MLLMs under the same setting, our HyViLM outperforms existing MLLMs in nine out of ten tasks. Specifically, HyViLM achieves a 9.6% improvement in performance on the TextVQA task and a 6.9% enhancement on the DocVQA task.

Figures

Figures reproduced from arXiv: 2412.08378 by the authors.

Figure 1
Figure 1. (a) An overall performance comparison between FILA and other existing MLLMs. (b) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An Overview of High-Resolution Methods. This Figure illustrates the differences between [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Whole architecture for MLLMs. (a) is the first stage of training, during which we only [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative Analysis. The issue of image fragmentation has been greatly reduced after [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results. The red solid line in the image represents the cropping boundary, and [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 4 canonical work pages

  1. [5]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500,

    arXiv

  2. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    arXiv 2010

  3. [7]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al

    URL https:// huggingface.co/datasets/laion/gpt4v-dataset. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multi- modal large language models. arXiv:2306.13394,

    arXiv

  4. [9]

    URL https://arxiv.org/abs/2312. 08914. Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. arXiv preprint arXiv:2409.03420,

    arXiv

  5. [11]

    Andreas K ¨opf, Yannic Kilcher, Dimitri von R ¨utte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich ´ard Nagyfi, et al

    URL https://arxiv.org/abs/2304.02643. Andreas K ¨opf, Yannic Kilcher, Dimitri von R ¨utte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich ´ard Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36,

    arXiv

  6. [12]

    Mini-gemini: Mining the potential of multi-modality vision language models, 2024a

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models, 2024a. URL https://arxiv.org/abs/2403.18814. Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resol...

    arXiv

  7. [13]

    Deepseek-vl: Towards real-world vision-language understanding, 2024a

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024a. URL https: //arxiv.org/abs/2403.05525. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh...

    arXiv

  8. [14]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque

    URL https: //arxiv.org/abs/2403.03003. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning,

    arXiv

Show all 29 references
  1. [15]

    Minesh Mathew, Viraj Bagal, Rub `en P´erez Tito, Dimosthenis Karatzas, Ernest Valveny, and C

    URL https://arxiv.org/abs/2203.10244. Minesh Mathew, Viraj Bagal, Rub `en P´erez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021a. URL https://arxiv.org/abs/2104.12756. Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A datase...

    2007 arXiv

  2. [16]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

    URL https://arxiv.org/abs/2103.00020. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models,

    arXiv

  3. [17]

    URL https://arxiv.org/abs/1910. 02054. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robe...

    1910

  4. [18]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach

    URL https://arxiv.org/abs/2210.08402. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read,

    arXiv

  5. [20]

    A Vaswani

    URL https://arxiv.org/abs/2302.13971. A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems ,

    arXiv

  6. [21]

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang

    URL https://arxiv.org/abs/2312.06109. Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images. arXiv preprint arXiv:2403.11703,

    arXiv

  7. [22]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    12 Published as a conference paper at ICLR 2025 Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arX...

    2025 arXiv

  8. [23]

    URL https://arxiv.org/abs/2303.15343. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridh...

    arXiv

  9. [24]

    org/abs/2205.01068

    URL https://arxiv. org/abs/2205.01068. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding,

    arXiv

  10. [25]

    Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang

    URL https://arxiv.org/abs/2306.17107. Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less is more: Focus attention for efficient detr. InProceedings of the IEEE/CVF international conference on computer vision, pp. 6674–6683,

    arXiv

  11. [26]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny

    URL https://arxiv.org/abs/2305.11206. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592,

    arXiv

  12. [27]

    We compared our model with Minigemini-HD and LLaV A-NeXT

    13 Published as a conference paper at ICLR 2025 A M ORE VISUALIZATION We selected images from DocVQA, WikiTableQuestions, and TextVQA, designed questions, and shifted the images so that the answers are at the edge of the cropped boundary. We compared our model with Minigemini-...

    2025

  13. [28]

    For the language model, we utilize LLaMA3-8B-Instruct (Touvron et al., 2023)

    as the high-resolution visual encoder and the main architecture of Hybrid Encoder, respectively. For the language model, we utilize LLaMA3-8B-Instruct (Touvron et al., 2023). We optimize the model for 1 epoch using the AdamW optimizer with a cosine learning rate schedule and a...

    2023

  14. [29]

    1 Published as a conference paper at ICLR 2025 C A LIGNMENT STRATEGY Conv StageInput Dimensions (D, H, W)Output Dimensions (D, H, W)ViT LayerViT Dimensions (D, H, W) 1 (192, 192,

    2 strategy, completing the optimization in a total of 32 hours on 32 × A800 GPUs. 1 Published as a conference paper at ICLR 2025 C A LIGNMENT STRATEGY Conv StageInput Dimensions (D, H, W)Output Dimensions (D, H, W)ViT LayerViT Dimensions (D, H, W) 1 (192, 192,

    2025

  15. [2018]

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi

    URL https://arxiv.org/abs/1801.08163. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV,

    arXiv

  16. [2019]

    Rub`en Tito, Dimosthenis Karatzas, and Ernest Valveny

    URLhttps://arxiv.org/ abs/1904.08920. Rub`en Tito, Dimosthenis Karatzas, and Ernest Valveny. Document collection visual question an- swering. In ICDAR 2021,

    1904 arXiv

  17. [2020]

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut

    URL https://arxiv.org/abs/2005.14165. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,

    2005 arXiv

  18. [2021]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang

    URL https: //arxiv.org/abs/2102.08981. Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models, 2024a. URL https://...

    arXiv

  19. [2022]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

  20. [2023]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al

    URL https: //arxiv.org/abs/2311.12793. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of ...

    arXiv

  21. [2024]

    URL https://arxiv.org/abs/2310.14566. 10 Published as a conference paper at ICLR 2025 Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual langu...

    2025 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.