Pith. sign in

REVIEW 5 major objections 7 minor 96 references

Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training

T0 review · 5 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a token-level reweighting of the captioning loss by text-only LLM surprisal consistently improves vision-language pre-training over naive next-token prediction, and that the gain grows with compute.

desk verdict A simple, controlled loss-reweighting method that likely helps in practice, but its stated image-relatedness mechanism is contradicted by the paper's own example and remains unvalidated. read the letter →

arxiv 2505.08971 v1 pith:HWKGJ7SM submitted 2025-05-13 cs.CV cs.CLcs.LG

Yangyi Chen , Hao Peng , Tong Zhang , Heng Ji This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords vision-languagepre-trainingnext-tokenpredictiontokenreweightingimportancesamplingtext-onlyreferencemodelimage-relatedtokensscalinglawshallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard next-token prediction is the wrong objective for vision-language pre-training because it spreads loss evenly over caption tokens, most of which have nothing to do with the picture. It proposes PRIOR, which reweights each token's loss by how surprising that token is to a text-only language model: tokens the text model cannot predict are treated as image-related and get upweighted. Across two architectures, encoder-based and encoder-free, this reweighted objective consistently outperforms the unweighted baseline by 19% and 8% average relative improvement, with better training stability and steeper scaling curves. If the effect holds, a purely offline, model-free weight can replace auxiliary objectives and data filtering in large-scale vision-language pre-training.

What carries the argument

The load-bearing object is the importance score $w_i = (1 - p_r(t_i|t_{<i}))^\alpha$, computed offline by a text-only reference model trained on captions without images. The paper inserts the normalized weight $k w_i / \sum_j w_j$ into the NTP loss so that tokens the text model finds hard to predict receive more gradient signal. This implements the paper's assumption that surprisal under a text-only model tracks visual information content; the same score drives the importance-sampling interpretation and the mutual-information justification in the appendix.

What would settle it

Annotate a sample of captions for human-rated image-relatedness and compute the correlation with $w_i = (1-p_r(t_i|t_{<i}))^\alpha$; if the correlation is not positive, or if an equal-strength reweighting with shuffled weights reproduces the gains, PRIOR's stated mechanism is not what drives the improvement.

Watch

Extended reading notes

Core claim

PRIOR's central claim is that the conditional probability assigned by a text-only LLM, $p_r(t_i|t_{<i})$, can identify which caption tokens carry visual information, and that reweighting the next-token-prediction loss by $w_i = (1 - p_r(t_i|t_{<i}))^\alpha$ makes vision-language pre-training more sample-efficient and less prone to hallucination. The paper frames this as importance sampling with a target distribution that upweights tokens the reference model finds surprising, then normalizes the weights across the caption and rescales by $k$ to preserve loss magnitude. The paper reports an 18.61% average relative gain for encoder-based LVLMs and 7.93% for encoder-free LVLMs at the end of 5,000 training steps, and higher scaling-law exponents $\alpha_D$ than plain NTP.

Load-bearing premise

The method stands or falls on the assumption that a token the text-only model struggles to predict is a token whose content comes from the image; the paper does not validate this mapping against human judgments of image-relatedness.

Editorial extensions

If this is right

  • Because the importance scores are computed offline and stored with the image-caption tuples, PRIOR can be dropped into existing NTP pipelines with minimal code changes and no extra reference-model inference during LVLM training.
  • The reweighting works for both encoder-based and encoder-free LVLM architectures, so it is not tied to a particular design choice about visual encoders.
  • PRIOR lowers loss on both image-related and image-unrelated tokens, which the paper reads as faster overall pre-training rather than a narrow gain on a single token class.
  • The reported scaling-law exponents imply PRIOR turns additional data and compute into downstream benchmark performance more efficiently than NTP.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader might infer that the same offline surprisal scores could be reused as a data-filtering or caption-rewriting signal, not just a loss weight, to clean noisy web-scale corpora before training.
  • The tokenizer-sharing limitation noted in the paper suggests a tokenizer-agnostic scoring scheme would make the weighted dataset portable across LVLM families.
  • The paper's own motivating example contains tokens such as a house address and price that are hard to predict from text yet are precisely the tokens the method intends to de-emphasize, so part of PRIOR's gain may come from variance reduction rather than from the stated image-relatedness story.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes PRIOR, a token-level reweighting of the next-token prediction (NTP) loss in vision-language pre-training. Each caption token receives a weight w_i = (1 - p_r(t_i | t_<i))^α, where p_r is the probability assigned by a text-only reference LLM trained on captions without images. The authors argue that tokens hard to predict from text alone are image-related, so upweighting them focuses pre-training on visual content. PRIOR is evaluated against standard NTP on two architectures—H-LVLMs (ViT-MLP-LLM) and U-LVLMs (unified transformer)—across six benchmarks, reporting 18.61% and 7.93% average relative improvements, improved training stability, and better scaling-law exponents. The paper also includes ablations on α and the loss-scale factor k, a comparison with ITM/ITC/ReC baselines, and an importance-sampling and mutual-information motivation.

Significance. If the empirical gains are robust, PRIOR is a simple, architecture-agnostic modification to the standard LVLM pre-training objective that requires only offline-computed token-level scores from a text-only LLM. The controlled comparison against NTP at multiple intermediate checkpoints on two architectures is a genuine strength: the gains are consistent in direction across most checkpoints, and the method is easy to integrate into existing pipelines. The paper does not yet ship code or release the reference-model scores, but the experimental setup is described in sufficient detail for replication. The central weakness is that the paper's stated mechanism—that hard-to-predict tokens are image-related—is not validated, and the headline improvement is computed in a way that overstates per-benchmark gains. The work is best viewed as an empirical contribution on surprisal-based reweighting, provided the authors either validate the mechanism or reframe the claims accordingly.

major comments (5)
  1. [§3.3, Figs. 2–3] The reported 'average relative improvement' of 18.61% for H-LVLMs is an aggregate relative gain (sum of PRIOR scores versus sum of NTP scores) rather than a mean of per-benchmark relative gains. From the final checkpoints in Fig. 2, the per-benchmark relative gains are approximately 19.7% (MME-P), 22.5% (MME-R), 12.4% (MMStar), 8.0% (POPE), 4.7% (MMBench), and 4.1% (SEEDBench), whose simple mean is about 11.9%. The aggregate number is dominated by the large-magnitude MME scores. Since the abstract and Fig. 1 headline the 19% figure, this is misleading; the authors should report per-benchmark gains or a clearly defined aggregated average.
  2. [§2.2, Eq. (3), Fig. 1] The load-bearing assumption that (1 - p_r(t_i | t_<i)) identifies image-related tokens is never validated. The paper's own motivating example in Fig. 1 describes tokens such as '7338 24th Ave NE, Seattle' and '$365,500' as image-unrelated, yet a text-only LLM assigns these tokens very low probability, so Eq. (3) upweights exactly those tokens rather than the highlighted visual words. Section 1 states that 100 human-annotated examples from CapsFusion are available, but no correlation between surprisal and human image-relatedness judgments is reported. The mutual-information argument in Appendix C does not repair this: the token-level MI contribution is log[p(t_i | v, t_<i) / p(t_i | t_<i)] (Eq. 14), and 1 - p_r is not a proxy for that log-ratio (for a fixed absolute probability gain, the log-ratio is largest when p_r is large, not small). The method may still improve downstream performance, but the paper does not demonstrate that it works by prioritizing image-related tokens. Please either provide the human-annotation validation or explicitly reframe the method as surprisal-based reweighting without the image-relatedness claim.
  3. [§4.1, Fig. 7] The importance scores are confounded with token position and caption length: Fig. 7 shows that the average importance score decreases with caption length (r = -0.243) and also decreases with token position within a caption. This means PRIOR changes the effective length and position distribution of the training loss relative to NTP. The observed gains could stem from these incidental properties rather than from the stated image-relatedness mechanism. A control experiment that reweights by position or length, or by a synthetic score with the same marginal distribution as the importance scores, would be needed to isolate the mechanism. As written, the comparison against NTP does not support the causal claim in the paper's title.
  4. [§2.3, Eq. (6)] The importance-sampling derivation is post-hoc: the target distribution in Eq. (6) is defined to be proportional to p_data(t_i | v, t_<i) * (1 - p_r(t_i | t_<i)), which makes the importance weight in Eq. (8) equal to (1 - p_r) by construction. This provides no independent support for the choice of weighting. Similarly, the statement that self-normalization (Eq. 9) 'substantially reduces variance' is asserted without any measurement or reference specific to this setting. The section should be presented as a framing or intuition rather than a derivation of the weight from first principles.
  5. [§3.5, Figs. 5–6] The scaling-law analysis lacks the statistical detail needed to support the claim of 'significantly higher scaling coefficients.' The number of data points per fit, the number of independent runs, and the definition of '8 sampling models' are not specified, and no confidence intervals or standard errors are reported for α_D. The extrapolation to 100M tokens from fits on data up to 70M tokens may be brittle, and the paper does not state whether the 100M-token point was actually evaluated. Please provide the fitted parameter values, uncertainty estimates, and a precise description of the sampling procedure.
minor comments (7)
  1. [§2.3, Eq. (7)] Equation (7) contains a typo: the subscript 'pdata(t_i|i,t_<i)' should read 'pdata(t_i|v,t_<i)'.
  2. [Abstract and §3.3] The abstract reports '19% and 8%' average relative improvement, while §3.3 reports 18.61% and 7.93%; these numbers should be made consistent.
  3. [§3.3, Figs. 2 and 9] The experimental curves appear to be from a single run per condition; error bars or a statement of the number of seeds would help assess the stability claim.
  4. [§3.5] The phrase 'including 8 sampling models' is undefined; please specify whether this refers to data subsets, checkpoints, or independent model runs.
  5. [§D (Limitations)] The limitation section acknowledges the tokenizer-sharing constraint but does not mention the unvalidated assumption behind Eq. (3); this missing validation should be listed as a limitation or addressed in the main text.
  6. [Fig. 1 caption] The caption states that highlighting is 'word-level for better visualization'; it would be clearer to specify how word-level annotations were mapped to the BPE tokens used for the probability scores.
  7. [§4.2, Fig. 8] The claim that 'PRIOR also optimizes learning on image-unrelated tokens' is based on average loss differences; lower loss on unweighted tokens may be an indirect effect of shared representations, so a causal interpretation requires more evidence.

Circularity Check

2 steps flagged · score 2.0 of 10

Minor post-hoc circularity in the importance-sampling motivation and the Fig. 8 consistency check; the central NTP-vs-PRIOR benchmark comparison is empirical, self-contained, and not circular.

  1. self definitional [Section 2.3, Eq. 6 and Eq. 8]
    "Target Distribution: We aim to define a new target distribution ptarget(ti|v,t<i) that assigns higher probability to tokens that are difficult for the reference model to predict: ptarget(ti|v,t<i)∝p data(ti|v,t<i)·(1−p r(ti|t<i)) ... Thus, the importance weight for each token loss ... is: w(ti|v,t<i) = ptarget(ti|v,t<i) / pdata(ti|v,t<i) ∝ (1−p r(ti|t<i))"

    The importance-sampling 'derivation' does not independently yield the weighting; the target distribution in Eq. 6 is defined as p_data scaled by (1-p_r), so the weight in Eq. 8 is the defining factor divided by p_data. The framework restates the design choice rather than justifies it. The central benchmark comparison is unaffected because the weights are not fit to any downstream benchmark.

  2. self definitional [Section 4.2, Fig. 8]
    "By applying various importance thresholds, we categorize tokens into image-related and image-unrelated subsets based on their assigned importance scores wi. We then measure the average NTP loss of H-LVLMs trained via two methods on these two distinct subsets. ... This monotonic relationship confirms that PRIOR delivers increasingly significant improvements for tokens with higher image relevance, validating its fundamental design principle of prioritizing tokens that carry the most visually related information."

    The 'image-related' subset is defined by the same score w_i that PRIOR's loss (Eq. 4) explicitly upweights, so the finding that PRIOR attains lower loss on high-w_i tokens is largely a consequence of the training objective rather than an independent validation. The analysis confirms that the reweighted objective reduces loss on the tokens it upweights, but it does not externally confirm that those tokens are image-related.

full rationale

Circularity pass. The headline result—PRIOR's 18.61% and 7.93% average relative gains over NTP—is an empirical comparison on MME, MMStar, POPE, MMBench, and SEEDBench. The token weights w_i=(1-p_r(t_i|t_<i))^alpha are computed offline from a text-only LLM (Eq. 3) and are not fitted to any benchmark, so the main result is not circular. Self-citations [14,15] are used for experimental setup (U-LVLM initialization, post-training data, scaling-law context) and are not load-bearing evidence for PRIOR's mechanism. Two passages do reduce to their own definitions. In Section 2.3, Eq. 6 defines the target distribution as p_data times (1-p_r), so Eq. 8's importance weight is that same factor by construction; presenting this as an importance-sampling derivation is a post-hoc restatement of the design choice. In Section 4.2/Fig. 8, the 'image-related' token groups are defined by the same w_i that PRIOR's loss upweights, so the reported lower loss on that subset is mostly a consistency check rather than external validation. Both issues are minor and do not affect the controlled benchmark comparison. Appendix C's mutual-information claim is asserted rather than derived, and the paper's own Fig. 1 example (address and price tokens are text-surprising yet image-unrelated) plus the length correlation in Fig. 7 raise a validity concern about the surprisal proxy; however, these are correctness concerns, not circularity reductions, so they do not raise the circularity score beyond 2.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger shows the method's load is carried by two hand or fit choices (alpha, scaling-law exponents) and by the unvalidated assumption that surprisal under a text-only LLM equals image-relatedness. No external entity is invented, but the central mechanism is an assumption rather than a derived fact.

free parameters (3)
  • alpha (importance exponent) = 1
    Set in Eq. 3 and chosen empirically; ablation in Fig. 10 shows 0.5-1 is stable and higher values (2, 4) degrade performance. This is a hand-tuned hyperparameter of the training objective.
  • alpha_D (scaling exponent, PRIOR and NTP) = not reported numerically
    Fitted to 8 sampled training runs in Eq. 11; the paper's scaling-superiority claim rests on these fitted exponents, but the text gives no point estimates, confidence intervals, or held-out 100M evaluation details.
  • D_c (scaling-law intercept, PRIOR and NTP) = not reported numerically
    Fitted jointly with alpha_D in Eq. 11; not reported in the text.
assumptions (5)
  • ad hoc to paper A token that is hard for a text-only LLM to predict is an image-related token.
    Core premise used in Eq. 3 and Fig. 1. No direct validation against human-labeled image-relatedness; the paper's Fig. 1 example contains high-surprisal, non-image tokens such as addresses and prices.
  • domain assumption The reference model's probability p_r(t_i|t_<i) approximates the true text-only conditional distribution p(t_i|t_<i).
    Used in Appendix C to connect the weighting to mutual information; the approximation error is not analyzed.
  • domain assumption Self-normalized importance sampling (Eq. 9) preserves the intended expected-loss reweighting despite known bias.
    Section 2.3 acknowledges self-normalization bias but provides no bound or empirical check.
  • domain assumption Downstream benchmark performance follows L(D) = (D/D_c)^alpha_D.
    Section 3.5 imports the LLM loss scaling-law form and applies it to VLM benchmark scores without justification.
  • domain assumption The sampled CapsFusion subsets (5M for reference LLM, 3M for weight computation) are representative of the full 120M dataset.
    Section 3.1 does not describe the sampling procedure; biased sampling could affect both the reference model and the training data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training." pith.science (2026). https://pith.science/paper/HWKGJ7SM

@misc{pith2026250508971,
  author       = {Pith},
  title        = {Pith review of: Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWKGJ7SM}},
  note         = {Machine review of arXiv:2505.08971}
}
read the original abstract

In standard large vision-language models (LVLMs) pre-training, the model typically maximizes the joint probability of the caption conditioned on the image via next-token prediction (NTP); however, since only a small subset of caption tokens directly relates to the visual content, this naive NTP unintentionally fits the model to noise and increases the risk of hallucination. We present PRIOR, a simple vision-language pre-training approach that addresses this issue by prioritizing image-related tokens through differential weighting in the NTP loss, drawing from the importance sampling framework. PRIOR introduces a reference model-a text-only large language model (LLM) trained on the captions without image inputs, to weight each token based on its probability for LVLMs training. Intuitively, tokens that are directly related to the visual inputs are harder to predict without the image and thus receive lower probabilities from the text-only reference LLM. During training, we implement a token-specific re-weighting term based on the importance scores to adjust each token's loss. We implement PRIOR in two distinct settings: LVLMs with visual encoders and LVLMs without visual encoders. We observe 19% and 8% average relative improvement, respectively, on several vision-language benchmarks compared to NTP. In addition, PRIOR exhibits superior scaling properties, as demonstrated by significantly higher scaling coefficients, indicating greater potential for performance gains compared to NTP given increasing compute and data.

Figures

Figures reproduced from arXiv: 2505.08971 by the authors.

Figure 1
Figure 1. (Top) Synthetic examples to highlight the motivation of PRIOR. Only a few tokens in the captions (highlighted in blue, word-level for better visualization) are related to the associated images. PRIOR utilizes probability scores from a text-only LLM to recalibrate the original loss function at the token level, prioritizing image-related tokens that receive lower probability scores from the LLM. (Bottom Left) PRIOR fo… view at source ↗
Figure 2
Figure 2. Main experimental results of LVLMs with pre-trained visual encoders. We compare PRIOR with the NTP vision-language pre-training across various training steps on LVLMs with pre-trained visual encoders, and we annotate the highest performance for each method, respectively. PRIOR demonstrates both superior performance and greater stability throughout the entire training. 3 Experiments 3.1 Implementation Details of PRIO… view at source ↗
Figure 3
Figure 3. The average performance compar￾ison on LVLMs with unified architectures. PRIOR demonstrates better performance and sta￾bility across the entire training process [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: The relative prediction error of NTP and PRIOR. We observe that the performance of LVLMs trained via PRIOR is more predictable at scale. NTP (Next-Token Prediction) PRIOR [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The scaling behavior comparison of NTP and PRIOR. PRIOR shows better scaling coefficients, indicating higher efficiency in translating increased resources into performance gains. The results presented in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Quantitative analysis of the importance score distribution. We find that the average importance score for each caption decreases (linearly) with the text length. Within each caption, the importance score decreases in later positions. scaling behavior, reflecting greate…
Figure 8
Figure 8. Figure 8: The comparison of NTP and PRIOR regarding the achieved loss on image-related and image-unrelated tokens. These two token groups are dynamically categorized based on varying the importance threshold wi (Eq. 3). We find that PRIOR accelerates the LVLMs training, consiste…
Figure 9
Figure 9. Figure 9: Main experimental results of LVLMs with unified architectures. We compare PRIOR with the NTP vision-language pre-training on LVLMs with unified architectures [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The ablation study of PRIOR regarding α and k on H-LVLMs. Results show optimal performance with α=1 and scaling factor k, justifying the design choices in PRIOR. A Additional Experimental Results The experimental results of LVLMs with unified architectures on all benc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 14 canonical work pages

  1. [1]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Singh Chap- lot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Th ´eophile Gervet, Soham Ghosh, Am´elie H´eliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Tim- oth´ee Lacroix, Guillaume Lample, Diego de Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo,...

    arXiv doi:10.48550/arxiv.2410.07073

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022

    arXiv 2022

  3. [3]

    MINT-1T: scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens

    Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Guha, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, Ran Xu, Yejin Choi, and Lud- wig Schmidt. MINT-1T: scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. In Amir Globersons, Lester Mackey, Danielle Bel- grave, Angela Fan, Ulrich Paquet, Ja...

    2024

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966, 2023. doi: 10.48550/ARXIV .2308.12966. URL https://doi.org/10.48550/arXiv.2308.12966

    arXiv doi:10.48550/arxiv.2308.12966 2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020

    2020

  6. [6]

    Lundberg, Harsha Nori, Hamid Palangi, Marco T´ulio Ribeiro, and Yi Zhang

    S´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco T´ulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, 2023

    2023

  7. [8]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and G¨ul Varol, editors, Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-Octo...

    work page doi:10.1007/978-3-031-7 2024

  8. [9]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

Show all 96 references
  1. [10]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Lin Bin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions. In Amir 10 Globerson...

    2024

  2. [11]

    Compcap: Improving multimodal large language models with composite captions

    Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh, Qifan Wang, David Yang, ShengYun Peng, Hanchao Yu, Shen Yan, Xuewen Zhang, and Baosheng He. Compcap: Improving multimodal large language models with composite captions. CoRR, abs/2412.05243,

    arXiv

  3. [12]

    Dress: Instructing large vision-language models to align and interact with humans via natural language feedback

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. arXiv preprint arXiv:2311.10081, 2023

    2023 arXiv

  4. [13]

    Vistruct: Visual struc- tural knowledge extraction via curriculum guided code-vision representation

    Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, and Heng Ji. Vistruct: Visual struc- tural knowledge extraction via curriculum guided code-vision representation. arXiv preprint arXiv:2311.13258, 2023

    2023 arXiv

  5. [14]

    Scaling laws for predicting downstream performance in llms

    Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms. arXiv preprint arXiv:2410.08527, 2024

    2024 arXiv

  6. [15]

    A single transformer for scalable vision- language modeling

    Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision- language modeling. CoRR, abs/2407.06438, 2024. doi: 10.48550/ARXIV .2407.06438. URL https://doi.org/10.48550/arXiv.2407.06438

    2024 doi arXiv

  7. [16]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  8. [17]

    Gonzalez, and Wei-Lin Chiang

    Christopher Chou, Lisa Dunlap, Koki Mashita, Krishna Mandal, Trevor Darrell, Ion Stoica, Joseph E. Gonzalez, and Wei-Lin Chiang. Visionarena: 230k real world user-vlm conversations with preference labels. CoRR, abs/2412.08687, 2024. doi: 10.48550/ARXIV .2412.08687. URL https:/...

    2024 doi arXiv

  9. [18]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, 2023

    2023

  10. [19]

    NVLM: open frontier-class multimodal llms

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: open frontier-class multimodal llms. CoRR, abs/2409.11402, 2024. doi: 10.48550/ARXIV .2409.11402. URL https://doi.org/10.48550/a...

    2024 doi arXiv

  11. [20]

    Unveiling encoder-free vision-language models

    Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, ed- itors, Advances in Neural Info...

    2024

  12. [21]

    A survey of vision-language pre-trained models

    Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. In Luc De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5436–5443...

    2022 doi

  13. [22]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.11691, 2024

    2024 arXiv

  14. [23]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony 11 Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston ...

    2024 doi arXiv

  15. [24]

    On pre-training of multimodal language models customized for chart understanding

    Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, and Leonid Sigal. On pre-training of multimodal language models customized for chart understanding. CoRR, abs/2407.14506,

    arXiv

  16. [25]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  17. [26]

    URL https://doi.org/10.48550/arXiv.2407

    doi: 10.48550/ARXIV .2407.14506. URL https://doi.org/10.48550/arXiv.2407. 14506

    doi arXiv

  18. [27]

    Making llama SEE and draw with SEED tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama SEE and draw with SEED tokenizer. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net,

    2024

  19. [28]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  20. [29]

    Gneissweb: Preparing high quality data for llms at scale

    Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah Adam, Abdulhamid Ade- bayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Borse, Yuan-Chi Chang, Xuan-Hong Dang, et al. Gneissweb: Preparing high quality data for llms at scale. arXiv preprint a...

    2025 arXiv

  21. [30]

    URLhttps://openreview.net/forum?id=0Nui91LBQS

  22. [31]

    Exploring the frontier of vision-language models: A survey of current methodologies and future directions

    Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. CoRR, abs/2404.07214, 2024. doi: 10.48550/ARXIV .2404.07214. URL https://doi.org/ 10.48550/arX...

    2024 doi

  23. [32]

    Infimm-webmath-40b: Advancing multi- modal pre-training for enhanced mathematical reasoning

    Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, and Quanzeng You. Infimm-webmath-40b: Advancing multi- modal pre-training for enhanced mathematical reasoning. CoRR, abs/2409.12568, 2024. doi: 10.48550/ARX...

    2024 doi arXiv

  24. [33]

    Multimodal-gpt: A vision and language model for dialogue with humans

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023

    2023 arXiv

  25. [34]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. CoRR, abs/2412.05237, 2024. doi: 10.48550/ARXIV .2412.05237. URL https://doi.org/10...

    2024 doi arXiv

  26. [35]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  27. [36]

    Compression represents intelligence linearly

    Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937, 2024

    2024 arXiv

  28. [37]

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, 12 Alex Paino, Ale...

    doi arXiv

  29. [38]

    The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer

    Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer. arXiv preprint arXiv:2504.10462, 2025

    2025 arXiv

  30. [39]

    Grounding language models to images for multimodal inputs and outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine L...

    2023

  31. [40]

    Rl with kl penalties is better viewed as bayesian inference

    Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022

    2022 arXiv

  32. [41]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  33. [42]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023

    2023 arXiv

  34. [43]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  35. [44]

    Silkie: Preference distillation for large visual language models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023

    2023 arXiv

  36. [45]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Pmlr, 2022

    2022

  37. [46]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. CoRR, 2023

    2023

  38. [47]

    Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text

    Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, Jiashuo Yu, Hao Tian, Jiasheng Zhou, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Zhenxi...

    2024 doi arXiv

  39. [48]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Mee...

    2024 doi

  40. [49]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. CoRR, 2019

    2019

  41. [50]

    Torchtitan: One-stop pytorch native solution for production ready LLM pre- training

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. Torchtitan: One-stop pytorch native solution for production ready LLM pre- training. CoRR...

    doi arXiv

  42. [51]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  43. [52]

    ´Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, and Zhiding Yu

    Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas V oegtle, Philipp Fischer, De-An Huang, Timo ...

    2025 doi arXiv

  44. [53]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  45. [54]

    Multimodal mamba: Decoder-only multimodal state space model via quadratic to linear distillation

    Bencheng Liao, Hongyuan Tao, Qian Zhang, Tianheng Cheng, Yingyue Li, Haoran Yin, Wenyu Liu, and Xinggang Wang. Multimodal mamba: Decoder-only multimodal state space model via quadratic to linear distillation. arXiv preprint arXiv:2502.13145, 2025

    2025 arXiv

  46. [55]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014

    2014

  47. [56]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR, 2023

    2023

  48. [57]

    Examining llms’ uncertainty expression towards questions outside parametric knowledge

    Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. Examining llms’ uncertainty expression towards questions outside parametric knowledge. arXiv preprint arXiv:2311.09731, 2023

    2023 arXiv

  49. [58]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023

    2023 arXiv

  50. [59]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  51. [60]

    Diving into self- evolving training for multimodal reasoning

    Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self- evolving training for multimodal reasoning. CoRR, abs/2412.17451, 2024. doi: 10.48550/ ARXIV .2412.17451. URLhttps://doi.org/10.48550/arXiv.2412.17451

    2024 doi arXiv

  52. [61]

    Mminstruct: A high-quality multi- modal instruction tuning dataset with extensive diversity

    Yangzhou Liu, Yue Cao, Zhangwei Gao, Weiyun Wang, Zhe Chen, Wenhai Wang, Hao Tian, Lewei Lu, Xizhou Zhu, Tong Lu, Yu Qiao, and Jifeng Dai. Mminstruct: A high-quality multi- modal instruction tuning dataset with extensive diversity. CoRR, abs/2407.15838, 2024. doi: 10.48550/ARX...

    2024 doi arXiv

  53. [62]

    DEEM: diffusion models serve as the eyes of large language models for image perception

    Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, Min Yang, and Binyuan Hui. DEEM: diffusion models serve as the eyes of large language models for image perception. CoRR, abs/2405.15232, 2024. doi: 10.48550...

    2024 doi arXiv

  54. [63]

    Deepseek-vl: Towards real-world vision-language understanding.CoRR, abs/2403.05525,

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding.CoRR, abs/2403.05525,

    arXiv

  55. [64]

    URL https://doi.org/10.48550/arXiv.2403

    doi: 10.48550/ARXIV .2403.05525. URL https://doi.org/10.48550/arXiv.2403. 05525. 14

    doi arXiv

  56. [65]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´e-Buc, Emily B. Fox, and Roman Garnett, ed- itors, Ad...

    2019

  57. [66]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021

    2021

  58. [67]

    Mmevol: Empowering multimodal large language models with evol-instruct

    Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, and Yongbin Li. Mmevol: Empowering multimodal large language models with evol-instruct. ...

    2024 doi arXiv

  59. [68]

    Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis

    Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv preprint ...

    2025

  60. [69]

    Policy opti- mization via importance sampling

    Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy opti- mization via importance sampling. Advances in Neural Information Processing Systems, 31, 2018

    2018

  61. [70]

    Laion- 5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Inform...

    2022

  62. [71]

    Shaker, Salman H

    Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. In IEEE/CVF Conference on Computer ...

    2024

  63. [72]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD ’20: The 26th ACM SIGKDD Confere...

    2020

  64. [73]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  65. [74]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, M...

    2024

  66. [75]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Aus...

    2018

  67. [76]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URLhttp://arxiv.org/abs/1909.08053. 15

    1909 arXiv

  68. [77]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023

    2023 arXiv

  69. [78]

    OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning...

    2022

  70. [79]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  71. [80]

    Hovle: Unleashing the power of monolithic vision- language models with holistic vision-language embedding

    Chenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang, Yu Qiao, et al. Hovle: Unleashing the power of monolithic vision- language models with holistic vision-language embedding. arXiv preprint arXiv:2412.16158, 2024

    2024 arXiv

  72. [81]

    Reconstructive visual instruction tuning

    Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. CoRR, abs/2410.09575, 2024. doi: 10.48550/ARXIV .2410.09575. URLhttps://doi.org/10.48550/arXiv.2410.09575

    2024 doi arXiv

  73. [82]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024 doi arXiv

  74. [83]

    Scaling pre-training to one hundred billion data for vision language models

    Xiao Wang, Ibrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai. Scaling pre-training to one hundred billion data for vision language models. arXiv preprint arXiv:2502.07617, 2025

    2025 arXiv

  75. [84]

    Simvlm: Sim- ple visual language model pretraining with weak supervision

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021

    2021 arXiv

  76. [85]

    Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4

    Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308.12067, 2023

    2023 arXiv

  77. [86]

    Knowledge-augmented few-shot visual relation detection

    Tianyu Yu, Yangning Li, Jiaoyan Chen, Yinghui Li, Hai-Tao Zheng, Xi Chen, Qingbin Liu, Wenqiang Liu, Dongxiao Huang, Bei Wu, and Yexin Wang. Knowledge-augmented few-shot visual relation detection. CoRR, 2023. 16

    2023

  78. [87]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 doi arXiv

  79. [88]

    Scaling text-rich im- age understanding via code-guided synthetic multimodal data generation

    Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, et al. Scaling text-rich im- age understanding via code-guided synthetic multimodal data generation. arXiv preprint arXiv:2502.14846, 2025

    2025 arXiv

  80. [89]

    Capsfusion: Rethinking image-text data at scale

    Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14022–14032, 2024

    2024

  81. [90]

    Omnialign-v: Towards enhanced alignment of mllms with human preference

    Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosong Cao, Weiyun Wang, Jiaqi Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, et al. Omnialign-v: Towards enhanced alignment of mllms with human preference. arXiv preprint arXiv:2502.18411, 2025

    2025 arXiv

  82. [91]

    Xing, Xiaodan Liang, and Zhiqiang Shen

    Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timo- thy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, and Zhiqiang Shen. Web2code: A large-scale webp...

    2024

  83. [92]

    Vinvl: Revisiting visual representations in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, ...

    2021

  84. [93]

    Pixel-sail: Single transformer for pixel-grounded understanding

    Tao Zhang, Xiangtai Li, Zilong Huang, Yanwei Li, Weixian Lei, Xueqing Deng, Shihao Chen, Shunping Ji, and Jiashi Feng. Pixel-sail: Single transformer for pixel-grounded understanding. arXiv preprint arXiv:2504.10465, 2025

    2025 arXiv

  85. [95]

    Minigpt-5: Interleaved vision-and-language generation via generative vokens

    Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language generation via generative vokens. CoRR, abs/2310.02239, 2023. doi: 10.48550/ARXIV .2310. 02239. URLhttps://doi.org/10.48550/arXiv.2310.02239

    2023 doi

  86. [96]

    Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. CoRR, 2023

    2023

  87. [97]

    Generalized decoding for pixel, image, and language

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In IEEE/CVF Conference on Computer Vision and...

    2023

  88. [2024]

    URL https://doi.org/10.48550/arXiv.2412

    doi: 10.48550/ARXIV .2412.05243. URL https://doi.org/10.48550/arXiv.2412. 05243

    doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.