Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A generative vision encoder with prompt-conditioned depth-breadth fusion outperforms CLIP-based encoders for multimodal LLMs on 25 benchmarks.

desk verdict Useful practical result: a single generative encoder (Florence-2) with multi-prompt, multi-depth features can beat CLIP and even multi-encoder systems, but the paper's 'controlled' comparison changes more than the encoder and should be read with that caveat. read the letter →

arxiv 2412.04424 v1 pith:NG3EV4WO submitted 2024-12-05 cs.CV cs.AI

Jiuhai Chen , Jianwei Yang , Haiping Wu , Dianqi Li , Jianfeng Gao , Tianyi Zhou , Bin Xiao This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsvisionencodergenerativepretrainingFlorence-2depth-breadthfusionvision-languagealignmentOCRinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that multimodal large language models are better built on a generative vision encoder than on the contrastive CLIP encoders that are currently standard. It uses Florence-2, which can be prompted to extract different kinds of visual information from the same image, and fuses those prompt-conditioned features with a lower-level visual stream using a channel-concatenation step called Depth-Breadth Fusion. The resulting models beat established baselines across 25 benchmarks, and a same-data comparison with LLaVA 1.5 isolates the encoder as the source of the gain. If the claim holds, a single multi-task generative encoder can replace the current practice of stitching together several specialized encoders, simplifying MLLM design while improving OCR, chart, and fine-grained perception performance.

What carries the argument

The central mechanism is Depth-Breadth Fusion (DBFusion): Florence-2 is asked to encode the same image under three task prompts (detailed caption, OCR, and dense region caption), producing three high-level token sets, and the raw DaViT encoder features are kept as a low-level stream; all four are concatenated along the channel dimension and passed through an MLP projection into the LLM. Channel concatenation keeps the token count at 576 while enriching each token, and the alignment-loss experiments show this fused representation aligns to LLM text embeddings better than CLIP, SigLIP, DINOv2, or Stable Diffusion features.

What would settle it

Compute the alignment loss for several encoders and also train each encoder in the full Florence-VL recipe on a fixed data budget; the paper's mechanism predicts that the ranking by alignment loss matches the ranking by final benchmark scores, so a single encoder with higher alignment loss that consistently outscores a lower-loss encoder would falsify the alignment-based explanation.

Watch

Extended reading notes

Core claim

The paper claims that the standard choice of CLIP-style contrastive encoders is suboptimal for multimodal LLMs: a generative vision model that can be prompted to produce task-specific features yields representations that align better with the LLM's text space. Using Florence-2 as the single vision encoder, with DBFusion to combine low-level DaViT features and high-level caption, OCR, and grounding features, Florence-VL outperforms established MLLMs, including models that fuse multiple separate encoders, across 25 benchmarks spanning general VQA, perception, hallucination, OCR, charts, and knowledge tasks, while using only 576 visual tokens. Under matched training data (the LLaVA 1.5 recipe), replacing CLIP with Florence-2 and DBFusion consistently raises scores, with the largest gains on OCR and chart/document understanding.

Load-bearing premise

The load-bearing premise is that the Section 4 alignment loss, a cross-entropy score between projected image features and text features, actually predicts downstream MLLM quality; if that proxy is unreliable, the paper's explanation of why Florence-2 works best is unsupported.

Editorial extensions

If this is right

  • Replacing a contrastive encoder with a generative, promptable encoder plus channel-concatenation fusion improves alignment and benchmark results under identical training data, so the encoder choice is a first-order factor in MLLM quality.
  • A single Florence-2 encoder can match or beat systems that mix multiple specialized encoders, saving the memory and latency of running several backbones.
  • The OCR-prompted feature is the main contributor to gains on text-heavy document and chart tasks; removing it degrades OCRBench, ChartQA, DocVQA, and InfoVQA.
  • Fusing the low-level DaViT feature with the high-level prompt-conditioned features is necessary; using only low-level or only high-level features raises alignment loss and lowers benchmark averages.
  • End-to-end pretraining of vision backbone, projector, and LLM, followed by finetuning of projector and LLM, is the recipe behind these results and is released in full.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The alignment-loss metric could be reused as a cheap, full-training-free screen for future vision encoders; if it is reliable, developers could rank candidate encoders in hours rather than weeks.
  • The experiments do not isolate whether Florence-2's advantage comes from its generative pretraining objective, its prompt-conditioned task diversity, or its DaViT architecture; a controlled study varying one factor at a time would separate these causes.
  • Since the features are selected by textual prompts, a routing layer that picks the OCR or grounding prompt on the fly based on the input image, or lets the LLM request a new visual feature mid-generation, is a natural extension the paper leaves open.
  • The OCR and chart gains suggest the same depth-breadth recipe could be pushed further by combining it with high-resolution tiling, which the paper notes is compatible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Florence-VL replaces CLIP-style contrastive vision encoders with Florence-2, a generative vision foundation model, and proposes Depth-Breadth Fusion (DBFusion) to combine features extracted from different encoder depths and from three task prompts (detailed caption, OCR, dense region caption). The fused features are channel-concatenated and projected into a pretrained LLM (Phi-3.5 or Llama-3). The model is trained with an end-to-end pretraining stage followed by partial fine-tuning on open-source data, and is evaluated on 25 benchmarks covering general VQA, vision-centric tasks, knowledge-based tasks, and OCR/Chart tasks. The paper also introduces an alignment-loss measurement to compare vision encoders and provides ablations on the depth and breadth components.

Significance. If the main empirical claims were cleanly established, this paper would be a useful contribution: it shows that a single generative vision encoder with prompt-conditioned features can rival or surpass multi-encoder systems such as Cambrian-8B, and it proposes a simple fusion method that does not increase the visual token count. The paper is also commendable for open-sourcing models and the full training recipe, and for attempting a matched-data comparison in Table 3. However, the central controlled comparison is confounded by a difference in the training recipe, and the alignment-loss analysis that motivates the mechanism is not validated against downstream task performance. The breadth and depth ablations show differences that are within noise. With revisions that address these issues, the contribution could be significant for the MLLM community.

major comments (3)
  1. [Section 6, Table 3 (vs. Supplementary Section 9)] The caption of Table 3 claims 'The key difference between them lies in the vision encoders used (CLIP for LLaVA vs. Florence-2 for our model), while we maintain the same training data and backbone LLMs for both.' This is contradicted by Supplementary Section 9, which states that during pretraining 'unlike LLaVA 1.5 which only tunes the projection layer, we fine-tune the entire model, including the vision backbone Florence-2, projection layer, and language model.' Since the authors themselves note that tuning the entire model yields better performance than freezing, the Table 3 comparison confounds the choice of vision encoder with a change in the pretraining recipe. The benchmark gains in Table 3 therefore cannot be attributed solely to Florence-2 features or DBFusion as claimed.
  2. [Section 4, Figures 4 and 5] The alignment-loss metric is computed by optimizing a trainable projection P to minimize a cross-entropy loss on the LLaVA-1.5 pretraining captions, but the paper never validates that this metric predicts downstream MLLM quality. No experiment shows that the ordering of encoders by alignment loss matches their ordering after full fine-tuning on standard benchmarks. Consequently, Figures 4 and 5 demonstrate only that Florence-2 features can be brought closer to frozen LLM text embeddings on the caption training distribution; they do not establish that lower alignment loss causes the benchmark improvements in Table 3, nor that the depth and breadth features are responsible for those improvements.
  3. [Table 5 (and Table 4)] The ablation study for breadth features reports average differences of less than one point across the benchmark suite, and several individual benchmarks improve when a feature is removed (e.g., removing the grounding feature improves GQA, VizWiz, MME-C, and Seed-image relative to the full model). Without error bars or significance tests, the claim that 'all high-level features are essential for maintaining optimal performance' is not supported. The same concern applies to the depth ablation in Table 4, where the full model outperforms [V] on most benchmarks but the margins are modest and unquantified.
minor comments (5)
  1. [Throughout] The paper consistently misspells 'breadth' as 'breath' (e.g., 'depth-breath fusion' in the title and Section 3.2), and contains typos such as 'LLama-3-8B-Instrcut', 'Cambrain', and 'Abalation' in Table 5 of the supplement.
  2. [Section 4, alignment loss equation] The definition of the alignment loss is unclear: the indicator matrix I(i,j)_n is not defined, the notation for the concatenated matrices F^v_n and F^t_n is inconsistent with the earlier per-pair notation f^v_n and f^t_n, and it is not stated whether the LLM is frozen when optimizing P. Please clarify the indexing and the optimization procedure.
  3. [Section 4] After average pooling, the feature matrices F^v_n and F^t_n are said to be in R^{N×d}, but the earlier definitions have vision tokens as R^{r_n×d'} and text tokens as R^{s_n×d}. The pooling and concatenation steps should be stated precisely so the dimensions are consistent.
  4. [References] HallusionBench and MMStar are evaluated but not cited in the reference list; please add the corresponding benchmark papers.
  5. [Tables 2 and 3] The main results and the matched-data comparison are reported without error bars or significance tests, so the repeated use of 'significant' in the abstract and Section 5 is not statistically substantiated. Reporting variances or a small number of repeated runs would strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark results are direct measurements, and the cited Florence-2 encoder is externally published; the Table 3 training-recipe mismatch is a confound, not a circular derivation.

full rationale

I examined the derivation chain from Florence-2 features through DBFusion to benchmark scores and found no step where a claimed prediction is equivalent to its input by construction. Florence-2 is cited as a published CVPR model [45] with external evaluation; although several Florence-VL authors are also Florence-2 authors, the present paper's contribution is the integration method (channel-concatenation DBFusion) and its benchmark performance is measured directly on external benchmarks such as MMBench, POPE, DocVQA, and ChartQA. No parameter is fitted to those benchmarks and then reported as a prediction. The alignment-loss analysis in Section 4 uses a trainable projection, but it is an independent diagnostic, not a fitted predictor of downstream scores; it supports, rather than defines, the architectural choice. The ablations in Tables 4-6 are direct comparisons. The only substantive concern is Section 6's claim that Table 3 isolates the vision encoder ('the key difference between them lies in the vision encoders used... while we maintain the same training data and backbone LLMs'), which is qualified by Supplementary Section 9: 'unlike LLaVA 1.5 which only tunes the projection layer, we fine-tune the entire model, including the vision backbone Florence-2, projection layer, and language model.' This is a real experimental confound that weakens the attribution of gains to Florence-2/DBFusion, but it is not circularity: the numbers are not derived from the model design by definition. No self-citation chain is invoked to forbid alternatives or force a conclusion. I therefore find no circular step and assign score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen prompts, an empirically selected fusion strategy, and a curated data recipe. No new physical or mathematical entity is introduced; the axioms are the reliability of Florence-2, the alignment-loss proxy, the benchmarks, and the training recipe.

free parameters (3)
  • Task prompt set (detailed caption, OCR, dense region caption) = Three fixed prompts
    Chosen by hand to cover context, text, and spatial relationships; no search or sensitivity analysis over alternative prompt sets is reported.
  • Fusion strategy (channel concatenation) = Channel integration
    Selected based on Table 1, where the gain over average pooling is about 0.4 average points; the choice is empirical, not derived.
  • Training data recipe = 16.9M captions + 10M instruction pairs
    Hand-curated from CC12M, Redcaps, Commonpool, PixelProse, Cambrian-7M, Vision Flan, ShareGPT4V, and Docmatix; no ablation isolates the contribution of each data component.
assumptions (4)
  • domain assumption Florence-2's prompt-based features are informative for downstream MLLM tasks.
    Relies on Florence-2's representation quality from [45] without independent verification in this paper beyond the reported benchmarks.
  • ad hoc to paper The cross-entropy alignment loss with a trainable projection measures vision-language alignment quality.
    Proposed in Section 4; no proof that lower alignment loss implies better downstream performance beyond the correlation shown in the paper.
  • domain assumption The 25 benchmarks used are reliable and comparable across models.
    Standard benchmarks are used, but no error bars, confidence intervals, or significance testing are provided.
  • domain assumption Whole-model pretraining followed by partial finetuning is necessary for the reported gains.
    The training recipe is stated in the appendix, but no ablation isolates the effect of this recipe choice versus other schedules.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion." pith.science (2026). https://pith.science/paper/NG3EV4WO

@misc{pith2026241204424,
  author       = {Pith},
  title        = {Pith review of: Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NG3EV4WO}},
  note         = {Machine review of arXiv:2412.04424}
}
read the original abstract

We present Florence-VL, a new family of multimodal large language models (MLLMs) with enriched visual representations produced by Florence-2, a generative vision foundation model. Unlike the widely used CLIP-style vision transformer trained by contrastive learning, Florence-2 can capture different levels and aspects of visual features, which are more versatile to be adapted to diverse downstream tasks. We propose a novel feature-fusion architecture and an innovative training recipe that effectively integrates Florence-2's visual features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted from different depths and under multiple prompts. Our model training is composed of end-to-end pretraining of the whole model followed by finetuning of the projection layer and the LLM, on a carefully designed recipe of diverse open-source datasets that include high-quality image captions and instruction-tuning pairs. Our quantitative analysis and visualization of Florence-VL's visual features show its advantages over popular vision encoders on vision-language alignment, where the enriched depth and breath play important roles. Florence-VL achieves significant improvements over existing state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks covering general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, etc. To facilitate future research, our models and the complete training recipe are open-sourced. https://github.com/JiuhaiChen/Florence-VL

Figures

Figures reproduced from arXiv: 2412.04424 by the authors.

Figure 1
Figure 1. Comparison of LLaVA-style MLLMs with our Florence [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of Florence-VL, which extracts visual features of different depths (levels of feature concepts) and breaths (prompts) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the first three PCA components: we apply PCA to image features generated from Detailed Caption, OCR, and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: We plot the alignment loss for different vision encoders, [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: We plot the alignment loss for various feature combi [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FastVLM: Efficient Vision Encoding for Vision Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A five-stage hybrid encoder that downsamples images 64x produces 4x fewer tokens than prior hybrid designs and 16x fewer than a ViT, giving about a 3x faster first-token time at similar VLM accuracy in Apple's benchmarks.

Reference graph

Works this paper leans on

49 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, 1https://huggingface.co/microsoft/Phi-3.5-mini-instruct 2https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXi...

    arXiv 2024

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 8

    arXiv 2023

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 5

    2023

  4. [4]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 5

    work page 2021

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions,

  6. [6]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 8

    arXiv 2024

  8. [8]

    Redcaps: Web-curated image-text data created by the people, for the people

    Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin John- son. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11431,

    arXiv

Show all 49 references
  1. [9]

    Davit: Dual attention vision transform- ers

    Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transform- ers. In European conference on computer vision, pages 74–

  2. [10]

    Mousi: Poly-visual-expert vision-language models

    Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, et al. Mousi: Poly-visual-expert vision-language models. arXiv preprint arXiv:2401.17221, 2024. 8

    2024 arXiv

  3. [11]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 5, 6

    2024

  4. [12]

    1 OCRBench ChartQA DocVQA InfoVQA Average Florence-VL 7B 41.4 24.3 44.5 29.4 34.9 OCR 40.9 22.9 44.4 29.0 34.2 (a) Ablation study on OCR features on OCR & Chart benchmark

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 1 OCRBench ChartQA DocVQA InfoVQA Average Florence-VL 7B 41.4 24.3 44.5 29.4 34.9 OCR 40.9 22.9 44.4 29.0 34.2 (a)...

    2024

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  6. [14]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  7. [15]

    Deciphering cross-modal alignment in large vision-language models with modality integration rate.arXiv preprint arXiv:2410.07167, 2024

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Deciphering cross-modal alignment in large vision-language models with modality integration rate.arXiv preprint arXiv:2410.07167, 2024. 4

    arXiv

  8. [16]

    Gqa: A new dataset for real-world visual reasoning and composi- tional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and composi- tional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. 5

    2019

  9. [17]

    https://huggingface.co/datasets/huggingfacem4/docmatix

    HuggingFaceM4/Docmatix. https://huggingface.co/datasets/huggingfacem4/docmatix. https : / / huggingface . co / datasets / HuggingFaceM4/Docmatix, 2024. 5

    2024

  10. [18]

    Brave: Broadening the visual encoding of vision-language models

    O ˘guzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. arXiv preprint arXiv:2404.07204, 2024. 8

    2024 arXiv

  11. [19]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–

    2016

  12. [20]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 1, 8

    2023

  13. [21]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 6

    2023 arXiv

  14. [22]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 5

    2023 arXiv

  15. [23]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  16. [24]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 6, 8

    2024

  17. [25]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 6, 8

    2024

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2, 4, 6, 7, 8

    2024

  19. [27]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 5

    2023 arXiv

  20. [28]

    On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2024. 6

    2024

  21. [29]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521,

  22. [30]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 6 2

    arXiv

  23. [31]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 6

    2022 arXiv

  24. [32]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 6

    2021

  25. [33]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. 6

    2022

  26. [34]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 5, 8

    2023 arXiv

  27. [35]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 5

    2022

  29. [37]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 8

    2022

  30. [38]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 1

    2024 arXiv

  31. [39]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6

    2019

  32. [40]

    From pixels to prose: A large dataset of dense image cap- tions, 2024

    Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels to prose: A large dataset of dense image cap- tions, 2024. 5

    2024

  33. [41]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 1, 2, 5, 6, 8

    2024 arXiv

  34. [42]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 6

    2024

  35. [43]

    Large language model evaluation via matrix entropy

    Lai Wei, Zhiquan Tan, Chenghai Li, Jindong Wang, and Weiran Huang. Large language model evaluation via matrix entropy. arXiv preprint arXiv:2401.17139, 2024. 4

    2024 arXiv

  36. [44]

    Grok 1.5v: The next generation of ai

    x.ai. Grok 1.5v: The next generation of ai. https://x. ai/blog/grok-1.5v, 2023. Accessed: 2024-07-26. 6

    2023

  37. [45]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818...

    2024

  38. [46]

    Vision-flan: Scaling human-labeled tasks in visual instruc- tion tuning, 2024

    Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruc- tion tuning, 2024. 5

    2024

  39. [47]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 5

    2023 arXiv

  40. [48]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  41. [49]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 8 3

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.