Pith. sign in

REVIEW 5 major objections 8 minor 2 cited by

From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs

T0 review · 5 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By adding a visual-embedding reconstruction loss to LLaVA-style pretraining, VDEP claims to improve image-text alignment and beat its baselines on most of 13 multimodal benchmarks.

desk verdict A simple, plausible auxiliary loss for MLLM pretraining, but the paper's central empirical claim is confounded by a doubled training budget and test-set hyperparameter selection. read the letter →

arxiv 2502.09093 v1 pith:E72XMMUO submitted 2025-02-13 cs.CV

Mingxiao Li , Fang Qu , Zhanpeng Chen , Na Su , Zhizhou Zhong , Ziyang Chen , Nan Du , Xiaolong Li This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsimage-textalignmentautoregressivepretrainingvisualembeddingreconstructionL2regressionlosshybridtrainingLLaVAVDEP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the alignment problem in multimodal large language models can be attacked from the image side during pretraining: instead of only recovering text from image-plus-text input, the model should also recover the image's own visual embedding from its hidden states. It packages this as VDEP, an auxiliary L2 reconstruction loss plus a hybrid alternating training schedule, and reports that this raises benchmark performance over the LLaVA baselines on 13 multimodal benchmarks without changing the architecture. A reader should care because the recipe is small and portable: if the claim holds, any LLaVA-style model can get stronger image-text alignment and higher visual question-answering scores by adding one loss and rebalancing pretraining data.

What carries the argument

The central object is the dynamic image embedding $\mathbf{X}_I$, the vector produced by the MLP that projects ViT patch features into the language model's input space. During VDEP mode, the paper uses that vector as a regression target for the LLM's corresponding hidden state $\mathbf{X}^h_I$, minimizing $\mathcal{L}_i = \|\mathbf{X}_I - \mathbf{X}^h_I\|_2^2$ under the claim that Euclidean closeness approximates maximizing mutual information $I(\mathbf{X}_I; \mathbf{X}^h_I)$. The hybrid schedule alternates VDEP mode with the standard LLaVA text-loss mode within each batch, controlled by a data-ratio hyperparameter, and substitutes a special `<auto image>` token in place of `<image>` during image-reconstruction training. This object carries the argument because it converts the image, which has no token labels, into the same 'predict the input' framing the text side already uses.

What would settle it

Train the same LLaVA pipeline with the same doubled data and the same alternating schedule, but replace the supervision target $\mathbf{X}_I$ with a fixed random vector of the same dimension; if the benchmark gains on OK-VQA and RealWorldQA persist, the visual content of the target is doing none of the work. Alternatively, estimate the mutual information $I(\mathbf{X}_I; \mathbf{X}^h_I)$ on a validation set: if the L2 loss drops while the mutual-information estimate does not rise, the stated link between the loss and information recovery is broken.

Watch

Extended reading notes

Core claim

The paper's central claim is that multimodal alignment in an MLLM should be treated as an information-recovery problem for both modalities, not just text. It proposes VDEP, which during pretraining takes the dynamic embedding $\mathbf{X}_I$ produced by the visual encoder's MLP and minimizes $\mathcal{L}_i = \|\mathbf{X}_I - \mathbf{X}^h_I\|_2^2$, where $\mathbf{X}^h_I$ is the hidden state of the LLM at the image-token positions. Under the paper's information-theoretic reading, driving this L2 difference to zero drives the conditional entropy $H(\mathbf{X}_I | \mathbf{X}^h_I)$ to zero and makes the hidden state carry the image's information. The paper reports that this hybrid objective, alternating VDEP mode with the standard LLaVA text-loss mode within each batch, outperforms the LLaVA baselines on most of 13 evaluated benchmarks and improves attention concentration on image content.

Load-bearing premise

The argument stands or falls on the claim that shrinking Euclidean distance between the MLP's image embedding and the LLM's hidden state at the same position captures semantic alignment, so that making that distance small improves downstream vision-language behavior rather than just compressing or distorting the hidden state.

Editorial extensions

If this is right

  • VDEP can be added to existing LLaVA-style MLLMs with no architectural changes, so any model in that family can inherit the alignment signal by appending one auxiliary loss and an alternating training mode.
  • On the paper's results, 7B LLaVA-VDEP gains +3.36 on OK-VQA and +2.84 on RealWorldQA, while 3B TinyLLaVA-VDEP gains +3.28 on VizWizQA, suggesting the benefit is not limited to one scale.
  • The hybrid objective shifts attention toward image tokens during pretraining, which the paper presents as reducing modality imbalance and hallucination.
  • The method adds roughly 3 hours of pretraining time because it doubles the number of training samples to let VDEP and standard LLaVA modes both receive enough data.
  • Ablations show performance is sensitive to the image-loss weight $\alpha$, with smaller values such as 0.001 working best at 7B scale, implying the balance between modalities matters as much as the loss itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: replacing the semantic target $\mathbf{X}_I$ with a frozen random projection of the image while keeping the same doubled data and alternating schedule would separate the benefit of extra compute and data from the benefit of the specific visual supervision content.
  • The objective would plausibly transfer to MLLMs with different projectors such as Q-Former or perceiver resamplers, since it only requires a hidden state at image positions and a target vector from the visual branch; that extension is not reported.
  • The information-theoretic framing implies a stronger check the paper does not run: if the loss genuinely raises $I(\mathbf{X}_I; \mathbf{X}^h_I)$, an estimator of that mutual information should rise on held-out images, and image information should be increasingly decodable from later LLM layers.
  • Because the largest gains land on knowledge-heavy and spatial benchmarks like OK-VQA and RealWorldQA, a natural follow-up is to measure whether VDEP changes answer distributions on hallucination-prone prompts rather than only average accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The manuscript proposes VDEP, a pretraining objective for multimodal LLMs that adds an L2 loss between the LLM's hidden states at image-token positions and the MLP output embeddings from the visual encoder. Training alternates between a VDEP mode and the standard LLaVA mode within each batch, using an <auto image> token in the VDEP branch. The authors offer an information-theoretic justification for the loss and report improved scores over TinyLLaVA and LLaVA-v1.5 across 13 benchmarks. They also ablate the image-loss weight, the VDEP/LLaVA data ratio, and the loss function. The central claims are that VDEP consistently outperforms the baselines and that L2 reconstruction approximates mutual-information maximization between image and hidden representations.

Significance. If the empirical claims were established, VDEP would be an attractive drop-in pretraining loss requiring no architectural change: it uses standard LLaVA components, evaluates with lmms-eval, and reports results at two model scales. The ablations of α and data ratio are the right kinds of experiments to run. However, the present manuscript does not establish the central claim. The comparison is confounded by a doubled training budget, no α=0 control, no multiple-seed uncertainty, and test-set hyperparameter selection. The theoretical derivation is circular as written. These issues are load-bearing, so the contribution cannot be assessed from the reported evidence. The paper ships no code, which further limits reproducibility of the exact data splits and training setups.

major comments (5)
  1. [§4 / Appendix D] Section 4 states 'We double the input data to ensure these two strategies receive fair and sufficient training', and Appendix D says 'we utilize a dataset with double the training samples of the original. As a result, the training time increases by around 3 hours.' This means VDEP is not compared with the baseline at equal budget. With the 1:1 batch split described in §3.3, the VDEP model receives additional gradient updates from image-supervision data that the baseline never sees, so every positive delta in Tables 1-3 could be explained by more training. The paper needs an equal-budget control: LLaVA trained on the same doubled data (or twice the steps), and VDEP with α=0 on the same data. Without those controls, the abstract claim that VDEP outperforms baselines is not established. The bullet in Section 1 claiming 'without requiring additional data or architectural modifications' also contradicts the implementation.
  2. [§3.2, Eqs. (5)-(9)] The information-theoretic derivation is circular. Eq. (6) defines the loss as ||XI - Xh_I||^2; Eq. (7) states that as the loss goes to zero, Xh_I = XI; Eq. (8) then concludes that H(XI|Xh_I) = 0 and I(XI;Xh_I) = H(XI). That conclusion is baked into the definition of the L2 loss rather than derived from any probabilistic model. Moreover, for a deterministic mapping Xh_I = f(XI), H(XI|Xh_I) = 0 without f being the identity, so conditional entropy reaching zero does not imply Xh_I = XI. No joint distribution over images and hidden states is specified, and no argument is given that Euclidean closeness in this high-dimensional embedding space corresponds to semantic alignment. This matters because the mutual-information framing is the paper's stated motivation.
  3. [§5.3, Tables 4-5; Appendix C, Tables 6-7] The hyperparameters α and data ratio are selected by checking performance on RWQA, MME-P, MMB-VQA, OK-VQA, and in the appendix AI2D, MM-Vet, MMMU, MMTB, GQA, VizWizQA, VQA-T, and SQA-I, which are the same test benchmarks used for the headline comparisons. This is test-set selection: the reported models are the best of several configurations evaluated on the test data, so the results are optimistically biased. The paper should select hyperparameters on a held-out validation set (or a separate development benchmark) and report the chosen values before test evaluation. This is not a cosmetic issue; Table 4 in particular shows monotonic improvement as α decreases, and α=0 is never measured.
  4. [Tables 1-2 / §5.1] The claim in §5.1 that VDEP 'consistently outperforms the baseline' is contradicted by the reported numbers. At 3B, MMBENCH-EN drops by 1.18, MMBENCH-CN by 3.20, and OCRB by 2.00; at 7B, MM-Vet drops by 0.50 and MMMU by 0.40. Several positive deltas are within run-to-run noise (GQA +0.38, VizWiz +0.03 at 7B). No seeds, confidence intervals, or significance tests are reported for any benchmark. With the data-budget confound in place, the mixed sign pattern makes it impossible to attribute the improvement to the VDEP loss. Multiple seeds for at least the headline benchmarks are needed.
  5. [§3.2 / §5.3] The L2 objective at image positions admits a degenerate solution: the LLM can minimize the loss by passing the input embedding through to its hidden state with little transformation, which would not constitute image-text alignment. The paper does not analyze hidden-state geometry (e.g., norm growth, directional similarity, or changes in attention after removing the loss) to rule out this collapse. The attention visualizations in Figures 1 and 4 are suggestive but are single examples and are not quantified. A control experiment comparing the proposed loss with a loss toward a random target, or with the target's gradient stopped, would help establish that semantic content rather than mere copyability drives the results.
minor comments (8)
  1. [Title] The title contains a typo: 'V ocabulary' should be 'Vocabulary'.
  2. [§3.3, Eq. (10)] The sentence after Eq. (10) says 'where Lt and Lv denote the loss functions for text and image modalities', but the equation uses Li for the image loss; the symbol should be consistent.
  3. [§4 / Appendix D] Section 4 says the pre-training and fine-tuning datasets are 'identical to those utilized in LLava-v1.5', but then says 'We double the input data' and Appendix D confirms additional training samples; please clarify whether the doubled data is a superset, a repeated sampling of the same data, or a different data collection.
  4. [Table 2 / Appendix B.2] The column 'OCRB' is used inconsistently: the appendix describes OCRB as an OCR benchmark citing Mishra et al. (2019), but OCR-VQA is also listed as a VQA dataset in Section 4; please clarify the relationship and use one consistent name.
  5. [§5.2] There is a typo: 'waaker' should be 'weaker'.
  6. [§3.1] Equations (1) and (2) write cross-entropy in terms of P(Xt) and P(Xp_t) without defining a token-level distribution; please make the notation precise or state that the expression is a token-level autoregressive factorization.
  7. [Appendix B.2] The description 'MM-Vet ... using GPT-4 as a benchmark' is imprecise; GPT-4 is used as an evaluation judge, not as the benchmark data itself.
  8. [Figure 2] The caption mentions 'Dynamic Text Hidden State', but this term is not defined or used in the text; please explain it or remove it.

Circularity Check

2 steps flagged · score 6.0 of 10

The information-theoretic derivation is definitional (L2 distance to the input embedding is defined, then zero loss is equated to 'mutual information maximized'), and the headline benchmark gains are partly selected by tuning alpha and data ratio on the same benchmarks that are then reported as evidence of outperformance.

  1. self definitional [Section 3.2, Eqs. 5-9 (especially Eqs. 6-8)]
    "To maximize mutual information, we minimize the difference between XI and X h I. Considering that cross-entropy is not well-suited for the measurement of similarity between vectors, this study adopts the L2 norm to quantify the Euclidean distance-based difference between two vectors. By minimizing the loss function Li, the hidden vector X h I converges toward the original image embedding vector XI. When this condition is satisfied, the conditional entropy H(XI |X h I ) approaches zero, and the mutual information I(XI ; X h I ) becomes equal to the entropy H(XI )."

    The proposed loss is defined as L2 distance to the input embedding XI, and the paper then 'derives' that zero loss implies XI = Xh_I, which implies zero conditional entropy and maximal mutual information. This is tautological: any loss of the form ||XI - Xh_I||^2 is minimized when the two vectors are equal, and equality trivially makes the conditional entropy zero. No joint distribution or probabilistic model is specified, so I(XI; Xh_I) and H(XI|Xh_I) are never computed; the claimed equivalence between minimizing L2 and maximizing mutual information is baked into the definition of the objective rather than established. The conclusion that the hidden state 'captures the semantic information' of XI is just a restatement of the chosen supervision target.

  2. fitted input called prediction [Section 4 (Baseline and Implementation) and Section 5.1, with Tables 4-5 compared to Tables 1-2]
    "We thoroughly tune the hyperparameters for different-scale base models and report the best performance. The results, summarized in Table 1, demonstrate that the VDEP series consistently outperforms the baseline across all benchmarks."

    The hyperparameters alpha and data ratio are selected using the same benchmark columns (RWQA, MME-P, MMB, OK-VQA) that are later reported as evidence in Tables 1-2. 'Report the best performance' means the headline result is the maximum over the tuning grid on the evaluation set itself. The claim that VDEP 'consistently outperforms' on those benchmarks is therefore a restatement of the selection criterion, not an independent prediction from a fixed, untuned method. This is fitted input presented as empirical support, and the absence of an alpha=0 control further prevents attribution of the gain to the proposed loss.

full rationale

The paper's central method is an auxiliary L2 loss between the MLP-projected image embedding XI and the LLM hidden state Xh_I. The claimed information-theoretic support in Section 3.2 is not a derivation: Eq. 6 defines Li as ||XI - Xh_I||^2, Eq. 7 states the minimizing limit gives Xh_I = XI, and Eq. 8 then concludes that H(XI|Xh_I) goes to zero and I(XI;Xh_I) equals H(XI). That is definitional: if the loss is squared distance to the input, then zero loss means equality, and equality trivially gives zero conditional entropy. No joint distribution is introduced, so the entropy and mutual-information quantities are never actually evaluated; the claim that minimizing L2 maximizes mutual information is imposed by the choice of objective, not shown. This is a genuine circular step in the paper's first-principles narrative. On the empirical side, Section 4 states that hyperparameters were tuned and the best performance reported, and the ablation tables select alpha and data ratio on the same benchmark sets that are later used to assert consistent outperformance. Thus the benchmark evidence is partly a re-reporting of the selection criterion rather than a prediction from a fixed method. The benchmark numbers are still external measurements, so the paper is not wholly circular; the circularity is concentrated in the Section 3.2 derivation and in the test-set hyperparameter selection, with additional missing controls (equal training budget, alpha=0) that are experimental weaknesses rather than circularity. Score 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on one fitted hyperparameter (alpha=0.001), one fitted scheduling knob (data ratio=1.0), and several unproved design assumptions about why regressing hidden states to input embeddings should improve alignment. The invented <auto image> token is a minor internal mechanism with no external handle.

free parameters (2)
  • alpha (image loss weight) = 0.001
    Selected by scanning 0.1, 0.01, 0.001 on RWQA, MME-P, MMBench, and OK-VQA; smaller alpha performs best (Table 4).
  • data ratio (VDEP vs LLava mode) = 1.0
    Selected by scanning 0.5, 0.8, 1.0 (Table 5); higher ratio gives better benchmark scores, so the model is trained entirely or predominantly in VDEP mode.
assumptions (4)
  • ad hoc to paper L2 loss between image hidden state and input embedding approximates maximizing mutual information I(XI; Xh_I).
    Asserted in Section 3.2 (Eqs. 5-8) without a probabilistic model; the argument reduces to 'if hidden state equals the embedding, mutual information is at its maximum', which is a definitional restatement.
  • ad hoc to paper The MLP output embedding is a valid target for supervising LLM image hidden states.
    Section 3.2 introduces this supervision target as a design choice; no evidence shows that matching the input projection improves semantic alignment.
  • domain assumption Text distribution must be stabilized first and the hybrid ratio controls modality balance.
    Section 3.3 claims that the text distribution needs to reach a stable state before image alignment; no analysis supports this scheduling principle.
  • domain assumption Image tokens outnumber text tokens and cause background overfitting when image loss is large.
    Invoked in Section 5.3 to explain alpha and data-ratio scans; no measurement of token counts or overfitting is provided.
invented entities (1)
  • <auto image> special token
    purpose: Replaces the standard <image> token when image data is used in VDEP autoregressive mode, to switch between the two training modes.
    A new vocabulary token introduced by the authors; its only role is internal to the training schedule and it carries no out-of-paper falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs." pith.science (2026). https://pith.science/paper/E72XMMUO

@misc{pith2026250209093,
  author       = {Pith},
  title        = {Pith review of: From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E72XMMUO}},
  note         = {Machine review of arXiv:2502.09093}
}
read the original abstract

While MLLMs perform well on perceptual tasks, they lack precise multimodal alignment, limiting performance. To address this challenge, we propose Vision Dynamic Embedding-Guided Pretraining (VDEP), a hybrid autoregressive training paradigm for MLLMs. Utilizing dynamic embeddings from the MLP following the visual encoder, this approach supervises image hidden states and integrates image tokens into autoregressive training. Existing MLLMs primarily focused on recovering information from textual inputs, often neglecting the effective processing of image data. In contrast, the key improvement of this work is the reinterpretation of multimodal alignment as a process of recovering information from input data, with particular emphasis on reconstructing detailed visual features.The proposed method seamlessly integrates into standard models without architectural changes. Experiments on 13 benchmarks show VDEP outperforms baselines, surpassing existing methods.

Figures

Figures reproduced from arXiv: 2502.09093 by the authors.

Figure 1
Figure 1. Layer-wise attention visualization of visual-to-instruction information flow. The example is derived from LLava-Bench (Liu et al., 2024a) and the query is ”Describe this photo in detail”. The visualization results demonstrate that VDEP significantly enhances the model’s ability to capture critical features in images, with particularly outstanding performance in identifying object boundaries. additional encoders (Xin… view at source ↗
Figure 2
Figure 2. The LLava-VDEP network architecture incorporates two distinct training modes. The VDEP mode performs supervised learning on image data, while the LLava mode is dedicated to supervised learning on text data. During batch training, a ratio parameter is used to control the proportional occurrence of these two modes within each batch, enabling an effective balance in the learning process. CLIP(Radford et al., 2021) empl… view at source ↗
Figure 3
Figure 3. Illustration of our VDEP derivation process. (a) Text Pre-training: Convert text into embeddings using tokenization. The LLM generates hidden states, which are processed by the LM head to produce predicted tokens. Compute cross-entropy loss with the original input.(b) Image Pre-training: Divide images into patches. Convert patches into embeddings using visual branches without real labels. These embeddings guide the … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Layer-wise attention visualization of visual-to-instruction information flow. Displayed from top to bottom are the attention heatmaps from LLava-v1.5-7B and LLava-v1.5-7B-VDEP, respectively. The example is derived from LLava-Bench (Liu et al., 2024b) and the query is ”…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MS-DETR: Towards Effective Video Moment Retrieval and Highlight Detection by Joint Motion-Semantic Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MS-DETR improves moment retrieval and highlight detection by disentangling motion and semantic video features, sharing task information between the two tasks, and training on generated auxiliary captions.

  2. Fine-Grained Zero-Shot Object Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    The authors define fine-grained zero-shot object detection, build a 1,432-species bird benchmark (FGZSD-Birds), and show their hierarchical MSHC detector outperforms prior ZSD models on that benchmark.

Reference graph

Works this paper leans on

60 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Aflalo, E., Stan, G. B. M., Le, T., Luo, M., Rosenman, S., Paul, S., Tseng, S.-Y., and Lal, V. Fivl: A framework for improved vision-language alignment. arXiv preprint arXiv:2412.14672, 2024

    work page Pith review arXiv 2024

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1 0 (2): 0 3, 2023

    arXiv 2023

  4. [4]

    P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R

    Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., and Yeh, T. Vizwiz. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), Apr 2010. doi:10.1145/1805986.1806020. URL http://dx.doi.org/10.1145/1805986.1806020

    arXiv 2010

  5. [5]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., and Elhoseiny, M. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023 a

    arXiv

  6. [6]

    Shikra: Unleashing multimodal llm's referential dialogue magic

    Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023 b

    arXiv 2023

  7. [7]

    Vision transformer adapter for dense predictions

    Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., and Qiao, Y. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534, 2022

    arXiv 2022

  8. [8]

    N., and Hoi, S

    Dai, W., Li, J., LI, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P. N., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 49250--49267. Curran Associate...

    work page 2023

Show all 60 references
  1. [9]

    Fini, E., Shukor, M., Li, X., Dufter, P., Klein, M., Haldimann, D., Aitharaju, S., da Costa, V. G. T., B \'e thune, L., Gan, Z., et al. Multimodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402, 2024

    2024 arXiv

  2. [10]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https://arxiv.org/abs/2306.13394

    2024 arXiv

  3. [11]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  4. [12]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision an...

    2024

  5. [13]

    Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 6700--6709, 2019

    2019

  6. [14]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pp.\ 235--251. Springer, 2016

    2016

  7. [15]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning, pp.\ 5583--5594. PMLR, 2021

    2021

  8. [16]

    A., et al

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123: 0 32--73, 2017

    2017

  9. [17]

    Lisa: Reasoning segmentation via large language model

    Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9579--9589, 2024

    2024

  10. [18]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023 a

    2023 arXiv

  11. [19]

    Lmms-eval: Accelerating the development of large multimodal models, March 2024

    Li, B., Zhang, P., Zhang, K., Pu, F., Du, X., Dong, Y., Liu, H., Zhang, Y., Zhang, G., Li, C., and Liu, Z. Lmms-eval: Accelerating the development of large multimodal models, March 2024. URL https://github.com/EvolvingLMMs-Lab/lmms-eval. * indicates equal contribution

    2024

  12. [20]

    Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 0 9694--9705, 2021

    2021

  13. [21]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp.\ 12888--12900. PMLR, 2022

    2022

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 b

    2023

  15. [23]

    A large-scale dataset towards multi-modal multilingual instruction tuning

    Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X., et al. A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 3, 2023 c

    2023 arXiv

  16. [24]

    X., and Wen, J.-R

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023 d

    2023 arXiv

  17. [25]

    Vila: On pre-training for visual language models

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26689--26699, 2024

    2024

  18. [26]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  19. [27]

    Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26296--26306, 2024 a

    2024

  20. [28]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  21. [29]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp.\ 216--233

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp.\ 216--233. Springer, 2025

    2025

  22. [30]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 0 2507--2521, 2022

    2022

  23. [31]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp.\ 3195--3204, 2019

    2019

  24. [32]

    Simple open-vocabulary object detection

    Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp.\ 728--755. Springer, 2022

    2022

  25. [33]

    K., and Chakraborty, A

    Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp.\ 947--952. IEEE, 2019

    2019

  26. [34]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  27. [35]

    Schulman, J., Zoph, B., Kim, C., Hilton, J., Menick, J., Weng, J., Uribe, J. F. C., Fedus, L., Metz, L., Pokorny, M., et al. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2 0 (4), 2022

    2022

  28. [36]

    Towards vqa models that can read

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8317--8326, 2019

    2019

  29. [37]

    Towards all-in-one pre-training via maximizing multi-modal mutual information

    Su, W., Zhu, X., Tao, C., Lu, L., Li, B., Huang, G., Qiao, Y., Wang, X., Zhou, J., and Dai, J. Towards all-in-one pre-training via maximizing multi-modal mutual information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15888--15899, 2023

    2023

  30. [38]

    Siamese image modeling for self-supervised vision representation learning

    Tao, C., Zhu, X., Su, W., Huang, G., Li, B., Zhou, J., Qiao, Y., Wang, X., and Dai, J. Siamese image modeling for self-supervised vision representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2132--2141, 2023

    2023

  31. [39]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  32. [40]

    Mllm can see? dynamic correction decoding for hallucination mitigation

    Wang, C., Chen, X., Zhang, N., Tian, B., Xu, H., Deng, S., and Chen, H. Mllm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.11779, 2024

    2024 arXiv

  33. [41]

    Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation

    Wang, J., Wang, Y., Xu, G., Zhang, J., Gu, Y., Jia, H., Yan, M., Zhang, J., and Sang, J. Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023 a

    2023 arXiv

  34. [42]

    K., Singhal, S., Som, S., et al

    Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2023

  35. [43]

    Internimage: Exploring large-scale vision foundation models with deformable convolutions

    Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ ...

    2023

  36. [44]

    Towards semantic equivalence of tokenization in multimodal llm

    Wu, S., Fei, H., Li, X., Ji, J., Zhang, H., Chua, T.-S., and Yan, S. Towards semantic equivalence of tokenization in multimodal llm. arXiv preprint arXiv:2406.05127, 2024

    2024 arXiv

  37. [45]

    Grok-1.5 vision preview, April 2024

    x.ai. Grok-1.5 vision preview, April 2024. URL https://x.ai/blog/grok-1.5v. Accessed: 2025-01-26

    2024

  38. [46]

    Mitigating object hallucination via concentric causal attention

    Xing, Y., Li, Y., Laptev, I., and Lu, S. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926, 2024

    2024 arXiv

  39. [47]

    Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning

    Xu, Z., Shen, Y., and Huang, L. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint arXiv:2212.10773, 2022

    2022 arXiv

  40. [48]

    An improved baseline for reasoning segmentation with large language model

    Yang, S., Qu, T., Lai, X., Tian, Z., Peng, B., Liu, S., and Jia, J. An improved baseline for reasoning segmentation with large language model. arXiv e-prints, pp.\ arXiv--2312, 2023

    2023

  41. [49]

    mplug-owl: Modularization empowers large language models with multimodality

    Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  42. [50]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi

    Ying, K., Meng, F., Wang, J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y., Liu, S., et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024

    2024 arXiv

  43. [51]

    Reformulating vision-language foundation models and datasets towards universal multimodal assistants

    Yu, T., Hu, J., Yao, Y., Zhang, H., Zhao, Y., Wang, C., Wang, S., Pan, Y., Xue, J., Li, D., et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:2310.00653, 2023 a

    arXiv

  44. [52]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023 b

    2023 arXiv

  45. [53]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  46. [54]

    A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., and Liu, Z

    Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., and Liu, Z. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024 a . URL https://arxiv.org/abs/2407.12772

    2024 arXiv

  47. [55]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., and Qiao, Y. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023

    2023 arXiv

  48. [56]

    From redundancy to relevance: Enhancing explainability in multimodal large language models

    Zhang, X., Shen, C., Yuan, X., Yan, S., Xie, L., Wang, W., Gu, C., Tang, H., and Ye, J. From redundancy to relevance: Enhancing explainability in multimodal large language models. arXiv preprint arXiv:2406.06579, 2024 b

    2024 arXiv

  49. [57]

    Bubogpt: Enabling visual grounding in multi-modal llms

    Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581, 2023

    2023 arXiv

  50. [58]

    H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al

    Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16793--16803, 2022

    2022

  51. [59]

    Tinyllava: A framework of small-scale large multimodal models

    Zhou, B., Hu, Y., Weng, X., Jia, J., Luo, J., Liu, X., Wu, J., and Huang, L. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289, 2024

    2024 arXiv

  52. [60]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.