Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Unified Multimodal Understanding via Byte-Pair Visual Encoding

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Byte-pair encoding of vector-quantized image patches, guided by a priority blending co-occurrence frequency with spatial consistency and trained with a three-stage curriculum, lets a discrete-token multimodal model match…

desk verdict Solid training-recipe paper on visual BPE tokens, but the claimed spatial-consistency mechanism is not implemented as described and the controlled comparison is missing. read the letter →

arxiv 2506.23639 v1 pith:3GXTHBB6 submitted 2025-06-30 cs.CV cs.AI

Wanpeng Zhang , Yicheng Feng , Hao Luo , Yijiang Li , Zihao Yue , Sipeng Zheng , Zongqing Lu This is my paper
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsbyte-pairencodingvisualtokenizationdiscretetokenrepresentationvectorquantizationcurriculumlearningvision-languageunderstandingspatialconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that byte-pair encoding (BPE), the tokenization method behind text language models, can be carried over to images in a way that lets a multimodal model work entirely with discrete tokens instead of a continuous visual encoder. The proposed pipeline vector-quantizes image patches, then merges frequently co-occurring and spatially consistent token pairs into a compact visual vocabulary, and trains the model with a three-stage curriculum schedule. If the claim is right, the modality gap and information bottleneck of CLIP-style encoders disappear because image and text share one token space. The key empirical evidence is that the 16K-vocabulary variant reaches 80.6 on VQAv2 and 72.1 on MMBench, nearly matching the continuous-embedding model VILA-1.5 (80.9 and 72.3), while the same model without the BPE vocabulary collapses to 54.3 and 38.2.

What carries the argument

The central mechanism is the priority-guided visual BPE vocabulary construction. Starting from an 8,192-entry VQ-GAN codebook, the algorithm iteratively merges adjacent token pairs into new tokens, scoring each pair with $P(a,b)=F(a,b)+\alpha S(a,b)$: $F$ is the normalized frequency of adjacent co-occurrence and $S$ is the average over occurrences of $\exp(-\|u_i - \bar{u}\|^2/2\sigma^2)$, where $u_i$ records the relative direction of the pair in each image. The merged tokens extend the vocabulary, and the process repeats until a target size is reached. A second load-bearing piece is the three-stage curriculum training: embeddings are aligned alone, then the first 25 percent of transformer layers are unfrozen, then all parameters, with data ratios shifting from foundation captions to instruction data across stages.

What would settle it

Train two BPE vocabularies on the same quantized corpus, one with $S(a,b)$ computed from true two-dimensional relative positions $(dx,dy)$ as equation 6 reads and one with the orientation-only version in Algorithm 2; if the vocabularies essentially coincide and downstream VQAv2 and MMBench scores are indistinguishable, then the priority term carries no position-dependent spatial information and the structural-encoding claim is unsupported.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that discrete visual tokens, when built with a priority that mixes co-occurrence frequency and spatial consistency, can carry enough two-dimensional structure for a transformer language model to perform vision-language understanding at the level of continuous-embedding systems. The priority function $P(a,b) = F(a,b) + \alpha S(a,b)$ scores each candidate token pair, with $F$ the normalized co-occurrence frequency and $S$ the average spatial similarity of the pair's relative offset across the training corpus; the highest-scoring pairs are merged into new vocabulary entries and their occurrences replaced. A three-stage training procedure then aligns the new visual embeddings with frozen language parameters, selectively fine-tunes the early transformer layers, and finally fine-tunes everything, using a curriculum that shifts from foundation captions to complex reasoning and instruction data. The paper reports that this combination closes most of the gap to continuous-embedding baselines on VQAv2, MMBench, MME-P, SciQA-IMG, POPE, and VizWiz, and that removing the BPE vocabulary causes a large performance drop.

Load-bearing premise

The load-bearing premise is that the spatial-consistency score in the priority function actually measures what it claims; as implemented in Algorithm 2, it records only whether a token pair is horizontal or vertical, not where in the image the pair occurs, so the asserted structural information may reduce to orientation consistency.

Editorial extensions

If this is right

  • Discrete token models can reach the performance level of continuous-embedding models on standard vision-language benchmarks without a CLIP-style encoder.
  • The BPE vocabulary itself, not just added parameters or training time, is the load-bearing component: deleting it drops VQAv2 from 80.2 to 54.3.
  • Vocabulary size should be matched to available data: the 16K vocabulary overtakes 8K only after more than 1.5× the standard training data, and is less efficient per unit cost.
  • The curriculum order and the unfreezing order both matter, with single-stage training falling 12.8 and 14.1 points behind on perception and reasoning averages.
  • Because all modalities live in one token space, the framework is positioned as a stepping stone toward unified autoregressive generation of images and text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because Algorithm 2 sets $u_i$ to $(0,1)$ for horizontal and $(1,0)$ for vertical adjacency, the spatial consistency term $S(a,b)$ as implemented measures orientation consistency, not true position-dependent structure; a direct re-implementation with position-aware offsets would test whether the claimed structural encoding changes the learned vocabulary.
  • Editorial inference: the framework's logic suggests a testable extension to video, where temporal adjacency can be treated as a third merge direction, predicting similar gains on video QA benchmarks.
  • Editorial inference: the many near-zero embeddings in the 16K variant indicate a utilization ceiling, implying that an adaptive vocabulary schedule—growing the token count only as data grows—could recover the scaling benefit at lower cost.
  • Editorial inference: if unified discrete tokens are the cause of the gains, hallucination rates under identical data and compute budgets should differ systematically from continuous-encoder models; comparing POPE-style scores with controlled training would isolate the tokenization effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a framework for training multimodal LLMs on discrete visual tokens obtained by applying byte-pair encoding (BPE) to VQ-GAN indices. It introduces a priority-guided vocabulary construction that combines co-occurrence frequency with a spatial consistency term, a curriculum-based data composition strategy, and a three-stage training procedure with progressive parameter unfreezing. The authors train Being-VL-0.5/0.5+ on top of Llama-3.1-8B and report competitive results on VQAv2, MMBench, MME-P, POPE, and VizWiz relative to continuous-embedding models, with an ablation showing a large drop when the BPE vocabulary is removed.

Significance. If the claims hold, the paper would provide evidence that discrete visual tokens can approach continuous-encoder performance in multimodal understanding, a valuable result for unified architectures. The within-model w/o-BPE ablation (80.2 to 54.3 on VQAv2, Table 1) is a strong demonstration that the BPE vocabulary matters. The paper also includes useful analyses of embedding weight distributions and scaling/efficiency trade-offs. However, the specific mechanism claimed to drive the improvement, the spatial consistency term in the priority function, is not actually implemented as described and is not isolated by ablation, substantially weakening the contribution.

major comments (3)
  1. [Appendix D, Algorithm 2 vs Section 3.4, Eq. (6)] The implementation of the spatial consistency term does not measure relative positioning. For each occurrence, u_i(a,b) is set to (0,1) for horizontal adjacency and (1,0) for vertical adjacency, discarding the actual coordinates (i,j) of the occurrence. Consequently S(a,b) measures whether a pair appears consistently in one orientation, not whether it appears consistently at a particular spatial location or with a particular spatial offset. The claim in Section 3.4 that 'token pairs with consistent spatial relationships across images receive higher scores' is therefore not supported by the algorithm as written.
  2. [Section 3.4, Eq. (5) with Algorithm 2] The combined priority P(a,b)=F(a,b)+alpha*S(a,b) is dominated by S in the implementation. F is normalized by the corpus-wide sum of pair counts (Algorithm 2 line 14), making typical F values very small probability masses, while S is an average of exponentials that lies in [0.94, 1.0] for sigma=2.0 given the two-orientation encoding. With alpha=0.3, alpha*S is on the order of 0.28-0.3, several orders of magnitude larger than F for most pairs, so the priority ranking effectively ignores frequency. Since the paper provides no ablation varying alpha or replacing S with a position-aware metric, the claimed frequency-spatial tradeoff is unverified.
  3. [Table 1, Section 4.2] The experimental evidence does not isolate the contributions of the priority-guided encoding. The only controlled ablation is 'w/o BPE', which removes the entire BPE vocabulary and thus changes token sequence length, vocabulary size, and training dynamics simultaneously; it does not test whether the spatial consistency term (or the priority function at all) is responsible for the gains over frequency-only BPE, which is the article's primary methodological novelty. The comparisons to published baselines also do not control for training data composition and compute, so the claim of being 'competitive with continuous embedding-based models' rests on uncontrolled comparisons.
minor comments (6)
  1. [Section A.1] alpha=0.3 and sigma=2.0 are said to be determined through ablation studies on a validation set, but no ablation table for these parameters is provided, so their optimality cannot be assessed.
  2. [Algorithm 2, line 30] The diversity filter applies a threshold tau, but neither the similarity measure nor the value of tau is defined anywhere in the paper; this is a missing hyperparameter for a step that is stated to be part of the method.
  3. [Section 4.1] The baselines are said to have 'similar training data scales', but no quantitative comparison of training data composition or volume is given; the appendix lists datasets for the proposed model but not for the baselines, hindering a fair assessment.
  4. [Section 4.3] The text refers to 'activation magnitude' during inference, but the visualization shows embedding weight values; these are different quantities and the wording should be reconciled.
  5. [Section 4.4 and Figure 4] The claim that the 16K vocabulary 'eventually outperforms' 4K/8K beyond 1.5x data appears to be based on an extrapolation; please state which data points are measured and which are inferred, and clarify the 'same checkpoint' description.
  6. [Appendix A.4] The subset of LAION-2B-en used as Foundation Data is not quantified and the selection criterion is not described, which impedes reproducibility of the data composition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: reported results rest on external benchmarks and ablations, not on a reduction of predictions to fitted inputs.

full rationale

The paper's central claims are empirical: Being-VL-0.5/+ scores on VQAv2, MMBench, MME-P, SciQA-IMG, POPE, and VizWiz are compared against external continuous and discrete baselines, and the w/o-BPE ablation isolates the BPE vocabulary. The priority function P(a,b)=F(a,b)+alpha*S(a,b) (Eq. 5) and spatial consistency S(a,b) (Eq. 6) are vocabulary-construction criteria; no downstream benchmark number is algebraically derived from these quantities, and the hyperparameters alpha=0.3 and sigma=2.0 are tuned on a validation set, which is standard and does not constitute predicting a fitted quantity. Citation [15] is to the same group's ICLR 2025 work, but it is used as background motivation and as the Being-VL-0 baseline; the present paper's comparison to Being-VL-0 and its own w/o-BPE variant provides independent evidence for the BPE design. The noted mismatch between Eq. 6's 'relative positioning' and Algorithm 2's orientation-only u_i in {(0,1),(1,0)} is a potential implementation/validation gap affecting whether S measures full spatial consistency, but it is a correctness concern rather than a circular reduction: the benchmark results are not defined in terms of S, and no fitted parameter is renamed as a prediction. No circular step satisfying the rubric's evidence standard is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method adds several hand-chosen hyperparameters and relies on cited theory and fixed pretrained components. No new physical or conceptual entities are introduced. The main burden is that the core spatial-consistency mechanism is implemented more weakly than described, and multiple unspecified algorithm constants would be needed to reproduce the vocabulary construction.

free parameters (6)
  • alpha (spatial consistency weight) = 0.3
    Controls balance of frequency F(a,b) and spatial consistency S(a,b) in priority score P=F+alpha*S (Eq. 5); set via validation ablation (Appendix A.1).
  • sigma (spatial kernel width) = 2.0
    Width in the similarity kernel exp(-||u1-u2||^2/(2 sigma^2)) (Eq. 7); validation-tuned.
  • tau (diversity filtering threshold) = not reported
    Algorithm 2 filters candidate pairs with similarity > tau to existing tokens but never defines tau or the similarity measure.
  • top-k (candidate pairs per iteration) = not reported
    Algorithm 1 selects top-k pairs by priority before choosing the argmax, but k is not specified.
  • curriculum data ratios per stage = Stage1 80/20/0/0; Stage2 40/30/20/10; Stage3 15/15/30/40
    Hand-chosen composition ratios (Table 7) are a core part of the proposed curriculum strategy.
  • early-layer fraction k in Stage 2 = 25% of layers
    Number of transformer layers unfrozen in selective fine-tuning; default without sensitivity analysis.
assumptions (4)
  • domain assumption Visual BPE reduces prediction loss for quantized two-dimensional data under the generative process analyzed in [15].
    The paper relies on this cited theorem to motivate the approach but does not re-derive or verify its conditions for natural images quantized by VQ-GAN.
  • ad hoc to paper The S(a,b) term as implemented in Algorithm 2, horizontal vs vertical orientation only, captures the spatial/structural consistency described in Eq. 6.
    Algorithm 2 sets u_i to (0,1) or (1,0) rather than actual image coordinates, so spatial consistency as implemented is orientation consistency; the paper does not flag this reduction.
  • domain assumption A curriculum from captions to perception to reasoning complements the hierarchical nature of BPE visual tokens.
    No evidence is provided that BPE token hierarchy maps to task complexity; the curriculum is motivated qualitatively in Section 3.6.1.
  • domain assumption VQ-GAN with an 8192-entry codebook preserves the visual information required by downstream language reasoning.
    Used as a fixed preprocessing stage (Section 3.1, Appendix A.1); quality loss from quantization is not evaluated against alternative codebook sizes or continuous features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unified Multimodal Understanding via Byte-Pair Visual Encoding." pith.science (2026). https://pith.science/paper/3GXTHBB6

@misc{pith2026250623639,
  author       = {Pith},
  title        = {Pith review of: Unified Multimodal Understanding via Byte-Pair Visual Encoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GXTHBB6}},
  note         = {Machine review of arXiv:2506.23639}
}
read the original abstract

Multimodal large language models (MLLMs) have made significant progress in vision-language understanding, yet effectively aligning different modalities remains a fundamental challenge. We present a framework that unifies multimodal understanding by applying byte-pair encoding to visual tokens. Unlike conventional approaches that rely on modality-specific encoders, our method directly incorporates structural information into visual tokens, mirroring successful tokenization strategies in text-only language models. We introduce a priority-guided encoding scheme that considers both frequency and spatial consistency, coupled with a multi-stage training procedure based on curriculum-driven data composition. These enhancements enable the transformer model to better capture cross-modal relationships and reason with visual information. Comprehensive experiments demonstrate improved performance across diverse vision-language tasks. By bridging the gap between visual and textual representations, our approach contributes to the advancement of more capable and efficient multimodal foundation models.

Figures

Figures reproduced from arXiv: 2506.23639 by the authors.

Figure 1
Figure 1. Overview of our framework. The upper part shows the BPE vocabulary construction process: [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our multi-stage training strategy. The bottom part shows multimodal data being [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Visualization of embedding weight distributions across three model variants. The heatmaps represent [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Scaling potential and training efficiency comparison for different vocabulary sizes (4K, 8K, 16K). [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

    cs.AI 2026-04 unverdicted novelty 8.0 of 10

    FeynmanBench is the first benchmark for evaluating multimodal LLMs on diagrammatic reasoning with Feynman diagrams, revealing systematic failures in enforcing physical constraints and global topology.

Reference graph

Works this paper leans on

74 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    A survey on multimodal large language models.arXiv preprint arXiv:2306.13549, 2023

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models.arXiv preprint arXiv:2306.13549, 2023

    arXiv 2023

  2. [2]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  3. [3]

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.arXiv preprint arXiv:2502.09560, 2025

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.arXiv preprint arXiv:2502.09560, 2025

    arXiv 2025

  4. [4]

    Multimodal machine learning: A survey and taxonomy

    Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018

    2018

  5. [5]

    How to bridge the gap between modalities: A comprehensive survey on multimodal large language model.arXiv preprint arXiv:2311.07594, 2023

    Shezheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, and Weimin Zhang. How to bridge the gap between modalities: A comprehensive survey on multimodal large language model.arXiv preprint arXiv:2311.07594, 2023

    arXiv 2023

  6. [6]

    Can MLLMs Perform Text-to-Image In-Context Learning?

    Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, and Kangwook Lee. Can mllms perform text-to-image in-context learning? arXiv preprint arXiv:2402.01293, 2024

    work page Pith review arXiv 2024

  7. [7]

    Vision transformer with quadrangle attention.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle attention.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    work page 2024

  8. [8]

    Unified language-vision pretraining with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669, 2023

    Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669, 2023

    arXiv 2023

Show all 74 references
  1. [9]

    Videoorion: Tokenizing object dynamics in videos.arXiv preprint arXiv:2411.16156, 2024

    Yicheng Feng, Yijiang Li, Wanpeng Zhang, Hao Luo, Zihao Yue, Sipeng Zheng, and Zongqing Lu. Videoorion: Tokenizing object dynamics in videos.arXiv preprint arXiv:2411.16156, 2024

    2024 arXiv

  2. [10]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  3. [11]

    Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022

    2022 arXiv

  4. [12]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024. 12

    2024 arXiv

  5. [13]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InProceedings of the IEEE/CVF Conference on Computer Vision and...

    2024

  6. [14]

    Unicode: Learning a unified codebook for multimodal large language models.arXiv preprint arXiv:2403.09072, 2024

    Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, and Zongqing Lu. Unicode: Learning a unified codebook for multimodal large language models.arXiv preprint arXiv:2403.09072, 2024

    2024 arXiv

  7. [15]

    From pixels to tokens: Byte-pair encoding on quantized visual modalities

    Wanpeng Zhang, Zilong Xie, Yicheng Feng, Yijiang Li, Xingrun Xing, Sipeng Zheng, and Zongqing Lu. From pixels to tokens: Byte-pair encoding on quantized visual modalities. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  8. [16]

    Neural machine translation of rare words with subword units.arXiv preprint arXiv:1508.07909, 2015

    Rico Sennrich. Neural machine translation of rare words with subword units.arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  9. [17]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  10. [18]

    Theoretical analysis of byte-pair encoding

    László Kozma and Johannes Voderholzer. Theoretical analysis of byte-pair encoding. arXiv preprint arXiv:2411.08671, 2024

    2024 arXiv

  11. [19]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017

    2017

  12. [20]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  13. [21]

    Vitae: Vision transformer advanced by exploring intrinsic inductive bias

    Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34, 2021

    2021

  14. [22]

    Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond.International Journal of Computer Vision, pages 1–22, 2023

    Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond.International Journal of Computer Vision, pages 1–22, 2023

    2023

  15. [23]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  16. [24]

    Emu: Generative pretraining in multimodality.arXiv preprint arXiv:2307.05222, 2023

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality.arXiv preprint arXiv:2307.05222, 2023

    2023 arXiv

  17. [25]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  18. [26]

    Deepseek-vl: Towards real-world vision-language understanding, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024

    2024

  19. [27]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  20. [28]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  21. [29]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  22. [30]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  23. [31]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023

    2023 arXiv

  24. [32]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  25. [33]

    From specific-mllm to omni-mllm: A survey about the mllms alligned with multi-modality.arXiv preprint arXiv:2412.11694, 2024

    Shixin Jiang, Jiafeng Liang, Ming Liu, and Bing Qin. From specific-mllm to omni-mllm: A survey about the mllms alligned with multi-modality.arXiv preprint arXiv:2412.11694, 2024

    2024 arXiv

  26. [34]

    A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions.Ieee access, 11:14804–14831, 2023

    Arnab Barua, Mobyen Uddin Ahmed, and Shahina Begum. A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions.Ieee access, 11:14804–14831, 2023

    2023

  27. [35]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. InProceedings of the Asian Conference on Computer Vision, pages 18–34, 2024

    2024

  28. [36]

    Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    2024 arXiv

  29. [37]

    Visual hallucinations of multi-modal large language models

    Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. Visual hallucinations of multi-modal large language models. arXiv preprint arXiv:2402.14683, 2024

    2024 arXiv

  30. [38]

    Cognitive mirage: A review of hallucinations in large language models.arXiv preprint arXiv:2309.06794, 2023

    Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models.arXiv preprint arXiv:2309.06794, 2023

    2023 arXiv

  31. [39]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  32. [40]

    Neural discrete representation learning.Advancesin neural information processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advancesin neural information processing systems, 30, 2017

    2017

  33. [41]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019

    2019

  34. [42]

    Investigating the effectiveness of bpe: The power of shorter sequences

    Matthias Gallé. Investigating the effectiveness of bpe: The power of shorter sequences. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1375...

    2019

  35. [43]

    Attention with markov: A framework for principled analysis of transformers via markov chains.arXiv preprint arXiv:2402.04161, 2024

    Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with markov: A framework for principled analysis of transformers via markov chains.arXiv preprint arXiv:2402.04161, 2024

    2024 arXiv

  36. [44]

    Language models still struggle to zero-shot reason about time series.arXiv preprint arXiv:2404.11757, 2024

    Mike A Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series.arXiv preprint arXiv:2404.11757, 2024

    2024 arXiv

  37. [45]

    Toward a theory of tokenization in llms.arXiv preprint arXiv:2404.08335, 2024

    Nived Rajaraman, Jiantao Jiao, and Kannan Ramchandran. Toward a theory of tokenization in llms.arXiv preprint arXiv:2404.08335, 2024

    2024 arXiv

  38. [46]

    Pixel-level bpe for auto-regressive image generation

    Anton Razzhigaev, Anton Voronov, Andrey Kaznacheev, Andrey Kuznetsov, Denis Dimitrov, and Alexander Panchenko. Pixel-level bpe for auto-regressive image generation. In Proceedings of the First Workshop on Performance and Interpretability Evaluations of Multimodal, Multipurpose...

    2022

  39. [47]

    Analyzing the language of visual tokens.arXiv preprint arXiv:2411.05001, 2024

    David M Chan, Rodolfo Corona, Joonyong Park, Cheol Jun Cho, Yutong Bai, and Trevor Darrell. Analyzing the language of visual tokens.arXiv preprint arXiv:2411.05001, 2024

    2024 arXiv

  40. [48]

    A semantic-based layer freezing approach to efficient fine-tuning of language models.arXiv preprint arXiv:2406.11753, 2024

    Jian Gu, Aldeida Aleti, Chunyang Chen, and Hongyu Zhang. A semantic-based layer freezing approach to efficient fine-tuning of language models.arXiv preprint arXiv:2406.11753, 2024

    2024 arXiv

  41. [49]

    Exploring selective layer fine-tuning in federated learning

    Yuchang Sun, Yuexiang Xie, Bolin Ding, Yaliang Li, and Jun Zhang. Exploring selective layer fine-tuning in federated learning. arXiv preprint arXiv:2408.15600, 2024

    2024 arXiv

  42. [50]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015. 14

    2015

  43. [51]

    A survey on multimodal benchmarks: In the era of large ai models.arXiv preprint arXiv:2409.18142, 2024

    Lin Li, Guikun Chen, Hanrong Shi, Jun Xiao, and Long Chen. A survey on multimodal benchmarks: In the era of large ai models.arXiv preprint arXiv:2409.18142, 2024

    2024 arXiv

  44. [52]

    A survey on benchmarks of multimodal large language models.arXiv preprint arXiv:2408.08632, 2024

    Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. A survey on benchmarks of multimodal large language models.arXiv preprint arXiv:2408.08632, 2024

    2024 arXiv

  45. [53]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  46. [54]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018

    2018

  47. [55]

    Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  48. [56]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  49. [57]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  50. [58]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  51. [59]

    Instructblip: towards general-purpose vision-language models with instruction tuning

    W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: towards general-purpose vision-language models with instruction tuning. arxiv.Preprint posted online on June, 15:2023, 2023

    2023

  52. [60]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  53. [61]

    mplug-owl: Modularization empowers large language models with multimodality, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models w...

    2023

  54. [62]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023

    2023

  55. [63]

    Hyperllava: Dynamic visual and language expert tuning for multimodal large language models, 2024

    Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, and Yueting Zhuang. Hyperllava: Dynamic visual and language expert tuning for multimodal large language models, 2024

    2024

  56. [64]

    Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023

    2023 arXiv

  57. [65]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023

    2023

  58. [66]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  59. [67]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InProceedings of ACL, 2018

    2018

  60. [68]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advancesin Neural Information Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advancesin Neural Informati...

    2022

  61. [69]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  62. [70]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. InEuropean conference on computer vision, pages 146–162. Springer, 2022

    2022

  63. [71]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  64. [72]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024

    2024

  65. [73]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558, 2024. 16 Appen...

    arXiv

  66. [74]

    • Reasoning Data (RD): We utilize 504K general QA entries and 343K reasoning-focused entries from the LLaVA-OneVision Dataset [71]

    to enhance detailed visual perception. • Reasoning Data (RD): We utilize 504K general QA entries and 343K reasoning-focused entries from the LLaVA-OneVision Dataset [71]. 17 Hyperparameter Stage 1 Stage 2 Stage 3 batch size 1 1 1 gradient accumulation 2 4 4 learning rate 1e-3 ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.