Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pisces is an 8B autoregressive model that reaches strong results on both image understanding and generation by giving each task its own visual encoder.

desk verdict Solid engineering paper, but the synergy claim rests on a confounded ablation and needs deconfounding before it can be taken literally. read the letter →

arxiv 2506.10395 v2 pith:E4CVLL6O submitted 2025-06-12 cs.CV cs.AI

Zhiyang Xu , Jiuhai Chen , Zhaojiang Lin , Xichen Pan , Lifu Huang , Tianyi Zhou , Madian Khabsa , Qifan Wang
show 5 more authors
Di Jin Michihiro Yasunaga Lili Yu Xi Victoria Lin Shaoliang Nie
This is my paper
classification cs.CVcs.AI
keywords unifiedmultimodalmodelautoregressiveimagegenerationunderstandingdecoupledvisualencodingcontinuoustokensGenEvalthree-stagepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pisces is an 8B autoregressive model trained to do both image understanding and image generation in one framework, and the paper's central claim is that a single model can be competitive with specialized systems on both. The key idea is that the two tasks want different visual representations: understanding needs many fine-grained visual tokens, while generation works much better with a short, pooled token sequence. To get both, Pisces uses a decoupled visual encoding architecture—one frozen encoder for understanding, a different frozen encoder for generation—and a three-stage training schedule that moves from short captions to detailed captions to instruction tuning. The reported results support the claim: best or near-best scores on over 20 understanding benchmarks, and a GenEval overall score of 0.65, the highest among the unified baselines compared. The paper also argues that the two tasks are synergistic, with each improving the other in a constrained-data comparison.

What carries the argument

The load-bearing object is the decoupled visual encoding architecture: two separate image-encoder paths into one shared multimodal language model. For understanding, the SigLIP encoder $\phi$ produces a long token sequence $V_n = \mathrm{MLPs}(\phi(I)) \in \mathbb{R}^{n \times d}$ that is prepended to the text; for generation, the gen-CLIP encoder $\varphi$ produces embeddings that are average-pooled with a $4 \times 4$ kernel down to $m = 64$ vectors $V_m = \mathrm{MLP}(\mathrm{Pool}_{4 \times 4}(\varphi(I))) \in \mathbb{R}^{m \times d}$, which the language model predicts autoregressively from the text prompt. The text side is trained with cross-entropy loss and the image-vector side with mean-squared error against the pooled encoder outputs, while the frozen encoders and a conditional latent-diffusion decoder (SDXL) handle reconstruction. This separation is what allows the same backbone to keep rich perceptual tokens for understanding and a short, learnable token sequence for generation.

What would settle it

A controlled experiment in which all variants are pretrained on the same image-caption data, changing only whether the understanding objective, the generation objective, or both are active, would settle the synergy claim: if the FID and benchmark gaps disappear when the data sources are matched, the claimed mutual benefit does not hold.

Watch

Extended reading notes

Core claim

The paper discovers that the long-standing underperformance of unified multimodal models is not primarily a capacity problem but an encoding problem: a single visual encoder and a single visual-token budget force the model to compromise between the dense semantic detail that understanding requires and the compact, autoregressively tractable representations that generation needs. Pisces therefore couples a SigLIP encoder producing a long sequence of visual tokens for understanding with a masked-autoencoding-trained CLIP encoder whose embeddings are pooled to 64 tokens for generation, both feeding one shared 8B language model, with a diffusion decoder that turns the predicted embeddings back into pixels. On this architecture the paper reports understanding results that match or beat specialized understanding models on most of the 20+ benchmarks tested, and generation results that surpass all prior open unified models on GenEval, with an overall score of 0.65. It further reports that joint training of understanding and generation improves both sides—understanding data lowers generation FID, and generation data raises understanding benchmark scores—and that the benefit depends on using different encoders for the two tasks.

Load-bearing premise

The synergy conclusion rests on a controlled comparison in which the with- and without-understanding variants also use different pretraining data sources (Shutterstock for generation-only, PixelProse for understanding-only), so it assumes that the data source itself does not drive the reported FID and benchmark gaps.

Editorial extensions

If this is right

  • A unified model need not sacrifice one capability for the other: the same 8B backbone can post 82.1 on VQAv2 and 0.65 on GenEval, beating larger unified systems on understanding and matching generation-only models in the same comparison table.
  • The architecture decouples token budgets as well as encoders: understanding can use long, high-resolution token sequences while generation stays cheap at 64 pooled vectors, so per-task inference cost can be set independently.
  • Joint training becomes a two-way data source: adding understanding data improves generation FID, and adding generation data improves understanding benchmarks, implying that data from either task can serve as auxiliary supervision for the other.
  • Detailed captions in the second pretraining stage drive better generation fidelity than the same volume of short captions, so caption richness is a tunable lever for unified generation quality.
  • The pooling-token count is a real hyperparameter: full-length 729-token targets raise the generation loss, 9 tokens lose too much detail, and the intermediate 64-token setting is the reported choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the synergy claim holds beyond the paper's comparison, a cheap diagnostic would be to cut understanding data from training and watch generation FID: a sharp regression would point to cross-task transfer as the source of the gain.
  • The decoupling recipe should transfer to other modality pairs, such as video understanding and generation, where the optimal encoder and token density for perception versus synthesis are likely to differ in the same way.
  • A practical extension would be to make the generation token budget adaptive—more pooled tokens for complex scenes, fewer for simple ones—since the paper shows both very long and very short token sequences hurt generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces Pisces, an 8B autoregressive multimodal model that shares a single LLM between image understanding and image generation while using two separate visual encoders: SigLIP for understanding and a MAE-trained CLIP variant (gen-CLIP) for generation. The two pathways use different numbers of visual tokens (a long sequence for understanding, a pooled 64-token sequence for generation), and a diffusion decoder reconstructs images from the generated embeddings. Training proceeds in three stages: 150M-image pretraining, 70M detailed-caption pretraining, and instruction tuning on 8M understanding instances plus 8M generation instances. The model is evaluated on more than 20 understanding benchmarks and on GenEval, reporting strong results (e.g., VQAv2 82.1, MMBench-EN 73.9, MM-Vet 50.0, GenEval overall 0.65) and claiming a synergistic relationship between understanding and generation based on the ablation in Section 6.1.

Significance. If the empirical claims hold, the work is practically significant: it demonstrates that a unified autoregressive model with decoupled visual encoders can reach competitive understanding performance and the highest GenEval score among the unified baselines in Table 2, while using a relatively small 8B LLM. The paper's strengths include an extensive and internally consistent benchmark evaluation, a clear architectural rationale for separating understanding and generation encoders, and concrete, falsifiable performance numbers. There is no circularity: the model is evaluated on external benchmarks and the design choices are empirical rather than definitional. The main limitations are the absence of error bars and contamination analysis, the use of only LLaVA 1.5 as specialized understanding baselines, and, most importantly, a confounded ablation for the synergy claim.

major comments (5)
  1. [§6.1, Table 3] The load-bearing claim that image understanding and generation are synergistic is not supported by the reported ablation. The three variants differ simultaneously in pretraining corpus (Shutterstock for 'Pisces w/o Und', PixelProse for 'Pisces w/o Gen', and an unspecified combination for 'Pisces'), caption style, image domain, and data quantity. The FID gap (38.2 vs. 78.4) and the understanding benchmark gaps (e.g., AI2D 64.1 vs. 60.7, SeedBench 74.4 vs. 68.4) could therefore be caused by data-source effects alone, independent of any cross-task synergy. In addition, the 'w/o Und' and 'w/o Gen' rows are each evaluated on only one task, so the mutual-benefit conclusion rests on asymmetric evidence. To establish synergy, the authors should hold the pretraining corpus fixed (or match data budgets and caption distributions) and vary only the task composition, then evaluate every variant on both understanding and generation benchmarks.
  2. [§2.2, Eq. (1)] Equation (1) is not a valid statement of the training objective. As written, it is a negative sum of next-token probabilities with no logarithm, no expectation over the data distribution, and unspecified summation bounds, and it does not include the MSE loss for image generation described in the surrounding text. Since this is the only formal equation for the unified objective, it should be corrected to a proper expected negative log-likelihood for text tokens plus an MSE term for image vectors, or replaced with separate objectives and a loss-weighting scheme.
  3. [§3.4 and §6.3] The final model's generation token configuration is ambiguous. Section 3.4 states that a 4x4 pooling kernel is used to obtain 64 gen-CLIP vectors, whereas Section 6.3 concludes by selecting 'pooling with a stride of 3' (which yields 81 tokens in the SigLIP ViT-L/16 setup) after comparing 729, 81, and 9 tokens. These two statements need to be reconciled, because the number of generation visual tokens is a central design parameter of the decoupled architecture.
  4. [§4.1 and §5.1, Table 1] The claim that Pisces 'surpasses models specifically designed for image understanding' is supported only by comparisons to LLaVA 1.5 7B and 13B. These are not contemporary specialized baselines, and no error bars or train/test contamination analysis are reported for the 20+ benchmarks. The extensive evaluation supports a claim of strong competitive performance, but the stronger statement about surpassing specialized models is not established with the current baseline set.
  5. [§6.2, Table 4] The ablation for decoupled visual encoders is incomplete on the generation side. The paper reports that using SigLIP for both understanding and generation yields inferior image reconstruction with SigLIP+SDXL, but no quantitative result is given for that arm; Table 4 only compares the decoupled model with the gen-CLIP-only model on understanding benchmarks. Without a measured generation FID or reconstruction error for the SigLIP-only variant, the claim that decoupling benefits generation is not directly demonstrated.
minor comments (7)
  1. [§2.2] The word 'lossely' should be 'loosely'.
  2. [§7] There are several typos in the Related Work section, including 'bootstrappning', 'utlize', 'discrepency', and 'yeilding'; these should be corrected.
  3. [§5.1] The text refers to 'Table 1b (a)' and 'Table 1b (b)', but the actual table is a single Table 1 with panels (a) and (b); the cross-references should be fixed.
  4. [§5.1] The reported improvements of '26.3% on MMBench' and '34.4% on MM-Vet' are relative improvements (e.g., (73.9-58.5)/58.5), not absolute percentage-point gains; the text should state this explicitly to avoid confusion.
  5. [§6.1] The description of the 'Pisces w/o Und' variant says it is pretrained exclusively on the Shutterstock dataset, but the full Pisces first stage uses Shutterstock images with short captions for generation and detailed Llama 3.2 captions for understanding; the exact data composition of each variant should be specified precisely.
  6. [§6.4, Table 5] The FID values in Table 5 (25.6, 27.6, 21.9) are not directly comparable to the FID value for Pisces in Table 3 (38.2) because the training settings differ (full 150M-stage-1 data vs. the 15M subset in §6.1); this should be clarified in the text.
  7. [General] No code, model checkpoints, or data release is mentioned; adding a reproducibility statement with release plans would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Pisces is evaluated against external benchmarks, and its architectural and training choices are supported by empirical ablations rather than by definitional fits or self-citation chains.

full rationale

The paper's core claims are empirical and externally benchmarked. Pisces reports scores on over 20 public understanding benchmarks and the GenEval generation benchmark using official evaluation code; these results are not produced by fitting a parameter and then re-predicting that same fitted quantity. The decoupled visual encoding architecture is motivated by pilot studies (Sections 6.2 and 6.3) and ablated with held-out comparisons, and the three-stage training recipe is justified by direct experiments such as the long-caption FID comparison in Table 5. No load-bearing mathematical uniqueness theorem or central premise is imported from the authors' prior work; self-citations such as Vision-Flan and MultiInstruct are used only as dataset or related-work references, not as justifications that force the paper's conclusions. The synergy claim in Section 6.1 is supported by a comparison of variants, and while that comparison is confounded because the variants differ in pretraining corpus (Shutterstock vs PixelProse) as well as task composition, a confounded experimental design is a validity concern, not a circular derivation: the table's numbers are measured outcomes, not consequences of definitions or fitted inputs. Overall, no step reduces by construction to its own inputs, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation is attempted; the contributions are empirical. The central claims rest on the listed design choices, pretrained components, and benchmark validity. No new physical or mathematical entities are introduced.

free parameters (4)
  • Generation visual token count = 64
    Chosen as an intermediate point after pilot comparing 729, 81, and 9 tokens (Fig. 4); 9 pools too aggressively and 729 is hard to learn. This value directly determines the generation architecture.
  • Pooling kernel and stride = 4x4
    Maps gen-CLIP features to 64 tokens; selected by hand based on reconstruction and loss experiments in Section 6.3.
  • Classifier-free guidance scale = not reported
    GenEval results depend on the CFG scale used at inference, but the paper does not state it, so the reported 0.65 cannot be reproduced externally.
  • Training data mixture sizes = 150M, 70M, 8M, 4M+4M
    Stage data volumes and ratios are hand-chosen and described as meticulous curation, with no ablation on mixture proportions.
assumptions (4)
  • domain assumption Pretrained LLaMA-3.1-Instruct 8B, SigLIP, gen-CLIP, and SDXL provide strong, transferable bases.
    The architecture freezes both image encoders and uses a pretrained LLM and diffusion decoder, assuming their representations and generative priors transfer to the unified objective (Sections 2 and 3.4).
  • domain assumption MSE regression against gen-CLIP embeddings is a learnable target for autoregressive image generation.
    Equation (1) treats continuous image vectors as tokens to predict; the paper does not prove this proxy is sufficient, only that it lowers training loss (Section 2.2).
  • domain assumption SDXL can decode pooled gen-CLIP embeddings into high-quality images.
    The diffusion decoder is trained and adapted on gen-CLIP embeddings; its reconstruction quality is assumed adequate for the GenEval results (Sections 2.3 and 6.2).
  • domain assumption External benchmarks are valid and uncontaminated by training data.
    No contamination analysis is provided for the 150M Shutterstock plus instruction-tuning data against public test sets like GenEval, VQAv2, or MMBench (Sections 3 and 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation." pith.science (2026). https://pith.science/paper/E4CVLL6O

@misc{pith2026250610395,
  author       = {Pith},
  title        = {Pith review of: Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E4CVLL6O}},
  note         = {Machine review of arXiv:2506.10395}
}
read the original abstract

Recent advances in large language models (LLMs) have enabled multimodal foundation models to tackle both image understanding and generation within a unified framework. Despite these gains, unified models often underperform compared to specialized models in either task. A key challenge in developing unified models lies in the inherent differences between the visual features needed for image understanding versus generation, as well as the distinct training processes required for each modality. In this work, we introduce Pisces, an auto-regressive multimodal foundation model that addresses this challenge through a novel decoupled visual encoding architecture and tailored training techniques optimized for multimodal generation. Combined with meticulous data curation, pretraining, and finetuning, Pisces achieves competitive performance in both image understanding and image generation. We evaluate Pisces on over 20 public benchmarks for image understanding, where it demonstrates strong performance across a wide range of tasks. Additionally, on GenEval, a widely adopted benchmark for image generation, Pisces exhibits robust generative capabilities. Our extensive analysis reveals the synergistic relationship between image understanding and generation, and the benefits of using separate visual encoders, advancing the field of unified multimodal models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transferability Between Understanding and Generation in Unified Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Cross-task capability transfer in UMMs is architecture-dependent and can be exploited by training understanding to improve generation while avoiding distribution shift.

Reference graph

Works this paper leans on

85 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    https://arxiv.org/abs/2407.21783

    The llama 3 herd of models, 2024. https://arxiv.org/abs/2407.21783

    arXiv 2024

  3. [3]

    CM3: A causal masked multimodal model of the internet

    Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet. CoRR, abs/2201.07520, 2022. https://arxiv.org/abs/2201.07520

    arXiv 2022

  4. [4]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar \' e n Simonyan

    Jean - Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Bin...

    2022

  5. [5]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    2023

  6. [7]

    Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- \( \) : Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net...

    2024

  7. [8]

    Deep compression autoencoder for efficient high-resolution diffusion models, 2024 b

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models, 2024 b . https://arxiv.org/abs/2410.10733

    arXiv 2024

  8. [9]

    Sharegpt4v: Improving large multi-modal models with better captions, 2023

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023. https://arxiv.org/abs/2311.12793

    arXiv 2023

Show all 85 references
  1. [10]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 c

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 c

    2024 arXiv

  2. [11]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, abs/2305.06500, 2023. doi:10.48550/ARXIV.2305.0650...

    2023 doi arXiv

  3. [12]

    Dream LLM : Synergistic multimodal comprehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dream LLM : Synergistic multimodal comprehension and creation. In The Twelfth International Conferen...

    2024

  4. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj \" o rn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 12873--12883. Computer Vision Foundation / IEEE , 2021. doi:...

    2021

  5. [14]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \" u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image sy...

    2024

  6. [15]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  7. [16]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  8. [17]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Sy...

    2023

  9. [18]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913, 2017

    2017

  10. [19]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608--3...

    2018

  11. [20]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \' a r, and Ross B. Girshick. Masked autoencoders are scalable vision learners. CoRR, abs/2111.06377, 2021. https://arxiv.org/abs/2111.06377

    2021 arXiv

  12. [21]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709, 2019

    2019

  13. [22]

    Unified language-vision pretraining in LLM with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, Di Zhang, Wenwu Ou, Kun Gai, and Yadong Mu. Unified language-vision pretraining in LLM with dynamic discrete visual tokenization. CoRR, abs/23...

    2023 doi

  14. [23]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pages 235--25...

    2016

  15. [24]

    Generating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on...

    2023

  16. [25]

    MIMIC-IT: multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. MIMIC-IT: multi-modal in-context instruction tuning. CoRR, abs/2306.05425, 2023 a . doi:10.48550/ARXIV.2306.05425. https://doi.org/10.48550/arXiv.2306.05425

    2023 doi arXiv

  17. [26]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023 b

    2023 arXiv

  18. [27]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. CoRR, abs/2407.07895, 2024 a . doi:10.48550/ARXIV.2407.07895. https://doi.org/10.48550/arXiv.2407.07895

    2024 doi arXiv

  19. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. 202: 0 19730--19742, 2023 c . https://proceedings.mlr.press/v202/li23q.html

    2023

  20. [29]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. CoRR, abs/2403.18814, 2024 b . doi:10.48550/ARXIV.2403.18814. https://doi.org/10.48550/arX...

    2024 doi arXiv

  21. [30]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023 d

    2023 arXiv

  22. [31]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \' a r, and C

    Tsung - Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \' a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tom \' a s Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision...

    2014 doi

  23. [32]

    Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models

    Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models...

    arXiv

  24. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744, 2023 b . doi:10.48550/ARXIV.2310.03744. https://doi.org/10.48550/arXiv.2310.03744

    2023 doi arXiv

  25. [34]

    Visual instruction tuning, 2023 c

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023 c . https://doi.org/10.48550/arXiv.2304.08485

    2023 doi arXiv

  26. [35]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  27. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  28. [37]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023 d

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023 d

    2023 arXiv

  29. [38]

    On the hidden mystery of ocr in large multimodal models, 2024 c

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2024 c

    2024

  30. [39]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 0 2507...

    2022

  31. [40]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  32. [41]

    Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. CoRR, abs/2306.09093, 2023. doi:10.48550/ARXIV.2306.09093. https://doi.org/10...

    2023 doi arXiv

  33. [42]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209, 2021

    2021

  34. [43]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub \`e n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697--1706, 2022

    2022

  35. [44]

    SDXL: improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M \" u ller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952, 2023. doi:10.48550/ARXIV.2307.01952. https://doi.org/10...

    2023 doi arXiv

  36. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and ...

    2021

  37. [46]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21: 0 140:1--140:67, 2020. https://jmlr.org/pa...

    2020

  38. [47]

    Hierarchical text-conditional image generation with CLIP latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06125, 2022. doi:10.48550/ARXIV.2204.06125. https://doi.org/10.48550/arXiv.2204.06125

    2022 doi arXiv

  39. [48]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \" o rn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 106...

    2022

  40. [49]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj\"orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684--10695, June 2022 b

    2022

  41. [50]

    Multimodal instruction tuning with conditional mixture of lora

    Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. Multimodal instruction tuning with conditional mixture of lora. CoRR, abs/2402.15896, 2024. doi:10.48550/ARXIV.2402.15896. https://doi.org/10.48550/arXiv.2402.15896

    2024 doi arXiv

  42. [51]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326, 2019

    2019

  43. [52]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. CoRR, abs/2406.06525, 2024 a . doi:10.48550/ARXIV.2406.06525. https://doi.org/10.48550/arXiv.2406.06525

    2024 doi arXiv

  44. [53]

    EVA-CLIP: improved training techniques for CLIP at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: improved training techniques for CLIP at scale. CoRR, abs/2303.15389, 2023 a . doi:10.48550/ARXIV.2303.15389. https://doi.org/10.48550/arXiv.2303.15389

    2023 doi arXiv

  45. [54]

    Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, abs/2307.05222, 2023 b . doi:10.48550/ARXIV.2307.05222. https://doi.org/10.48550/arXiv.2307.05222

    2023 doi arXiv

  46. [55]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)...

    2024

  47. [56]

    Codi-2: In-context, interleaved, and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context, interleaved, and interactive any-to-any generation. CoRR, abs/2311.18775, 2023. doi:10.48550/ARXIV.2311.18775. https://doi.org/10.48550/arXiv.2311.18775

    2023 doi arXiv

  48. [57]

    Any-to-any generation via composable diffusion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 2024

    2024

  49. [58]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024

    2024

  50. [59]

    Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer

    Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, Hongsheng Li, Yu Qiao, and Jifeng Dai. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. CoRR, abs/2401.10208, 2...

    2024 doi arXiv

  51. [60]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024 a

    2024 arXiv

  52. [61]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568--9578, 2024 b

    2024

  53. [62]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient fo...

    2023 doi arXiv

  54. [63]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  55. [64]

    To see is to believe: Prompting GPT-4V for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu - Gang Jiang. To see is to believe: Prompting GPT-4V for better visual instruction tuning. CoRR, abs/2311.07574, 2023 a . doi:10.48550/ARXIV.2311.07574. https://doi.org/10.48550/arXiv.2311.07574

    2023 doi arXiv

  56. [65]

    OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Kamalika Chaudhuri, Stefanie Jegelka, Le Son...

    2022

  57. [66]

    Image as a foreign language: BEIT pretraining for vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEIT pretraining for vision and vision-language tasks. In IEEE/CVF Conference on Computer Vi...

    2023

  58. [67]

    Emu3: Next-token prediction is all you need, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tie...

    2024 arXiv

  59. [68]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024. https://arxiv.org/abs/2410.13848

    2024 arXiv

  60. [69]

    Next-gpt: Any-to-any multimodal LLM

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat - Seng Chua. Next-gpt: Any-to-any multimodal LLM . CoRR, abs/2309.05519, 2023. doi:10.48550/ARXIV.2309.05519. https://doi.org/10.48550/arXiv.2309.05519

    2023 doi arXiv

  61. [70]

    Grok 1.5v: The next generation of ai

    x.ai. Grok 1.5v: The next generation of ai. https://x.ai/blog/grok-1.5v, 2023. Accessed: 2024-07-26

    2023

  62. [71]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. CoRR, abs/2408.12528, 2024. doi:10.48550/ARXIV...

    2024 doi arXiv

  63. [72]

    Llava-uhd: an LMM perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat - Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an LMM perceiving any aspect ratio and high-resolution images. CoRR, abs/2403.11703, 2024 a . doi:10.48550/ARXIV.2403.11703. https://doi.org...

    2024 doi arXiv

  64. [73]

    Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning

    Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. In Anna Rogers, Jordan L. Boyd - Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023 doi

  65. [74]

    Vision-flan: Scaling human-labeled tasks in visual instruction tuning, 2024 b

    Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning, 2024 b . https://arxiv.org/abs/2402.11690

    2024 arXiv

  66. [75]

    Modality-specialized synergizers for interleaved vision-language generalists

    Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Modality-specialized synergizers for interleaved vision-language generalists. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, Apr...

    2025

  67. [76]

    Retrieval-augmented multimodal language modeling

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen - Tau Yih. Retrieval-augmented multimodal language modeling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, a...

    2023

  68. [77]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodal...

    2023 doi arXiv

  69. [78]

    LAMM: language-assisted multi-modal instruction-tuning dataset, framework, and benchmark

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, and Wanli Ouyang. LAMM: language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. CoRR, abs/2306.06687, 2023. doi:10...

    2023 doi arXiv

  70. [79]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang - Wen Li, Susan Zhang,...

    2023 doi arXiv

  71. [80]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023 b

    2023 arXiv

  72. [81]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  73. [82]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 11941--11952. IEEE , 2023. doi:10.1109/ICCV51070.2023....

    2023

  74. [83]

    Anygpt: Unified multimodal LLM with discrete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal LLM with discrete sequence modeling. CoRR, abs/2402.12226, 2...

    2024 doi arXiv

  75. [84]

    Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024. https://arxiv.org/abs/2408.11039

    2024 arXiv

  76. [85]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592, 2023 a . doi:10.48550/ARXIV.2304.10592. https://doi.org/10.48550/arXiv.2304.10592

    2023 doi arXiv

  77. [86]

    VL-GPT: A generative pre-trained transformer for vision and language understanding and generation

    Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. VL-GPT: A generative pre-trained transformer for vision and language understanding and generation. CoRR, abs/2312.09251, 2023 b . doi:10.48550/ARXIV.2312.09251. https://do...

    2023 doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.