Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By re-training a causal vision-language model with joint text-and-image masked reconstruction, MoCa converts it into a bidirectional embedding model that sets a new state of the art on the MMEB benchmark.

desk verdict MoCa's two-stage recipe is real and its MMEB result is strong, but the 'consistently outperforms all strong baselines' claim is false as written, and the missing causal-7B control muddies the main attribution claim. read the letter →

arxiv 2506.23115 v1 pith:DNKSUGZV submitted 2025-06-29 cs.CV cs.AIcs.CL

Haonan Chen , Hong Liu , Yuping Luo , Liang Wang , Nan Yang , Furu Wei , Zhicheng Dou This is my paper
classification cs.CVcs.AIcs.CL
keywords multimodalembeddingscontinualpre-trainingbidirectionalattentionmaskedlanguagemodelingautoencodingcontrastivefine-tuningvisionmodelsdocumentretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal embedding models usually inherit the causal attention of the vision-language backends they are built on, even though bidirectional encoders are known to make better embeddings. MoCa asks whether a causal VLM can be converted into a bidirectional embedding model with unlabeled data, and the paper answers yes. The first stage re-trains the VLM with a joint denoising objective that reconstructs masked text tokens and masked image patches from the surrounding interleaved multimodal context. The second stage contrastively fine-tunes the resulting bidirectional model on heterogeneous pairs—long documents, curated multimodal pairs, and text-only pairs—and the framework sets a new state of the art on MMEB: the 7B model averages 71.5, beating the previous best 11B model at 69.8.

What carries the argument

The load-bearing device is the joint reconstruction objective of the continual pre-training stage. For text tokens it applies masked language modeling, predicting masked tokens from the full bidirectional context; for image patches it applies masked autoencoding, reconstructing Gaussian-masked patches with a lightweight decoder conditioned on the surrounding image and text. The two losses share a single masked input, forcing the model to reason across modalities while its attention pattern switches from causal to bidirectional. The second stage's task-aware batching, which groups same-task instances so in-batch negatives are hard, is the supporting mechanism that makes contrastive fine-tuning effective on heterogeneous data.

What would settle it

Train a variant of MoCa in which the continual pre-training uses a next-token prediction loss instead of masked language modeling and masked autoencoding, but with the same bidirectional attention, unlabeled data, and contrastive fine-tuning; if that variant matches or exceeds MoCa's MMEB scores, the joint-reconstruction objective is not the cause of the gains and the central claim would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that masked reconstruction on massive unlabeled interleaved data is what unlocks the benefit of bidirectional attention for multimodal embeddings. Simply removing the causal mask in Qwen-2.5-VL and contrastively fine-tuning the result lowers MMEB accuracy at 3B (65.4 versus 66.4 for the causal variant), so the architecture change alone is not the source of the gains. After continual pre-training with masked language modeling and masked autoencoding on roughly 30B tokens of text, image-caption, and document-level data, the bidirectional model beats the causal one, and scaling from 3B to 7B parameters improves MMEB across every task category. The paper also claims that heterogeneous contrastive fine-tuning—long-form query-document pairs, curated multimodal pairs, and text-only pairs with task-aware batching—improves generalization beyond image-caption training.

Load-bearing premise

The paper's gains hinge on the transfer from masked reconstruction on unlabeled multimodal data to downstream retrieval quality, since simply switching to bidirectional attention without that pre-training hurts performance.

Editorial extensions

If this is right

  • A 7B bidirectional model can beat an 11B causal model on MMEB, meaning the architecture and training recipe, not raw parameter count, drive the gains.
  • Unlabeled interleaved data, much more abundant than curated image-caption pairs, can serve as the scaling resource for multimodal embeddings.
  • The 3B model's strong ViDoRe-v2 result suggests the recipe transfers to document-level retrieval and is not overfit to MMEB's task mix.
  • The method is presented as backbone-agnostic, so any causal VLM could potentially be converted with the same two stages.
  • The steady MMEB improvement as CPT corpus size grows supports the paper's claim of reliable data scalability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An iso-compute comparison, matching total training FLOPs or parameter count between MoCa and mmE5, could change the reported rankings; the current comparison confounds model size, training data, and data curation.
  • The joint MLM and MAE objective may be replaceable by any bidirectional denoising task; the paper does not isolate whether the reconstruction type or simply the bidirectional training signal is the essential ingredient.
  • The gap between in-distribution and out-of-distribution MMEB averages narrows with scale (10.8 points at 3B to 7.1 points at 7B), suggesting the method improves distribution coverage as it scales.
  • The same two-stage recipe could extend to video or audio tokens, since MLM and MAE generalize to any tokenized modality, which would test whether reconstruction-based bidirectional adaptation is a general principle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MoCa, a two-stage framework that converts pre-trained causal vision-language models (Qwen-2.5-VL) into bidirectional multimodal embedding models. Stage one performs modality-aware continual pre-training with a joint masked-language-modeling and masked-autoencoding objective on interleaved text-image data. Stage two applies contrastive fine-tuning on heterogeneous pairs (long-form documents, curated multimodal pairs, and text-only pairs) with task-aware batching. Experiments on MMEB and ViDoRe-v2 report new state-of-the-art results for MoCa-7B on MMEB and competitive performance on ViDoRe-v2, together with ablations, scaling curves, and hyperparameter sensitivity analyses.

Significance. If the claims are borne out, MoCa offers a practical recipe for adapting generative VLMs to bidirectional embedding use, and the emphasis on unlabeled interleaved data is a useful counterpoint to label-hungry contrastive approaches. The paper is strong in experimental breadth: it ablates each objective and data type, reports scaling behavior in Figure 3, and selects hyperparameters on validation sets rather than test sets. The method is clearly specified in Equations (1)--(3), and the two-stage design is easy to reproduce in principle. However, the headline claim of consistent outperformance is not supported by the paper's own tables, and the causal-versus-bidirectional comparison is confounded by the choice of fine-tuning data, so the central contribution needs careful revision.

major comments (3)
  1. [Section 3.2, Tables 1 and 2] The sentence "Our model, MoCa, consistently outperforms all strong baselines on both benchmarks" is contradicted by the reported numbers. In Table 2, MoCa-7B obtains an average NDCG@5 of 58.8 on ViDoRe-v2, below GME-7B's 59.3, and MoCa-3B's 59.8 exceeds GME-7B by only 0.5 points. In Table 1, MoCa-3B (67.5) is below mmE5-11B (69.8). Thus neither model size consistently outperforms all strong baselines on both benchmarks. The abstract's "new state-of-the-art results" is accurate only for MoCa-7B on MMEB. The claims should be reworded to state precisely which model variant achieves which result, and the discussion in the Introduction and Section 3.2 should be adjusted accordingly.
  2. [Section 3.2, Table 1] The contribution of bidirectional attention is not isolated. Table 1 shows that removing the causal mask without continual pre-training lowers MMEB average at 3B (65.4 vs. 66.4), and the bidirectional-with-CL baseline is trained only with contrastive learning on the MMEB training set, not with the full heterogeneous fine-tuning data. The paper's observation (1) compares "bidirectional + CPT + CL" against "bidirectional + CL" but does not provide a "causal + heterogeneous CL" or "causal + CPT + heterogeneous CL" control with the same data and objective mixture. Consequently, the improved scores attributed to the bidirectional architecture could in principle be driven by the CPT objectives and the broader contrastive data rather than by the attention change. Please add such controls at least at 3B (and ideally at 7B) or explicitly state the confound and soften the causal-attention claim.
  3. [Section 3.4, Figure 3] The scaling claim "after approximately 2,200 steps ... the 3B model achieves performance on par with the 7B baseline trained without CPT" is difficult to verify because the identity of the "7B Baseline" in Figure 3 is unspecified. The text does not say whether this is the 67.1 bidirectional-attention-7B variant from Table 1, the 66.4 causal-attention-3B variant, or some other checkpoint. Please identify the exact baseline and, if possible, include the corresponding curve in the figure so the comparison is unambiguous.
minor comments (5)
  1. [Tables 1 and 2 and surrounding text] The benchmark name is spelled inconsistently as "ViDoRe-v2" and "Vidore-v2" (Table 3 and Section 3.3); please unify.
  2. [Section 3.2, Table 2] The table header mixes task abbreviations (ESG_Human, Eco_Mul, Bio, etc.) without a legend; a brief explanation of each column would improve readability.
  3. [Section 3.1.2] The paper says each positive pair has two hard negatives, but Section 2.3 defines the loss with a general K; please state explicitly that K=2 in all experiments.
  4. [General] No error bars or significance tests are reported. Since several headline differences are small (e.g., 0.5 points on ViDoRe-v2), at least a statement about single-run versus multi-run evaluation would help calibrate confidence.
  5. [Abstract and Section 1] The Introduction's contribution bullet says the framework "achieves state-of-the-art performance on MMEB" without mentioning the ViDoRe-v2 result; aligning the abstract, introduction, and conclusion with the qualified findings would improve internal consistency.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: MoCa's claimed results are externally measured benchmark accuracies, with only minor non-load-bearing self-citations to the authors' own mmE5/E5 datasets.

full rationale

MoCa's central claims are benchmark accuracies on MMEB and ViDoRe-v2 obtained by training a two-stage pipeline and evaluating on held-out test sets; no quantity claimed as a prediction or state-of-the-art result is derived from the measured target by construction. The continual pre-training objective (MLM in Eq. 1 plus MAE in Eq. 2) and the contrastive loss (Eq. 3) are standard denoising and alignment losses, and the reported gains are empirical effects of applying them, not algebraic identities. Hyperparameter choices (mask ratios, MAE loss weight, learning rate, temperature) are stated to be selected on validation sets ('We select hyperparameter values based on performance on validation sets, each containing 1K samples drawn from the corresponding training data'), with final numbers reported on the MMEB test set, so there is no fitted-input-renamed-as-prediction pattern. The paper does cite works from its own author group: mmE5 [5] and E5 [39] supply fine-tuning data, and mmE5-11B is the strongest MMEB baseline. This is standard dataset reuse and head-to-head comparison, not a load-bearing self-citation chain: no uniqueness theorem or prior self-authored result is invoked to forbid alternatives or force the design choice, and the bidirectional-attention motivation is supported by external text-embedding literature (LLM2Vec [4], NV-Embed [21], etc.). The statement that MoCa 'consistently outperforms all strong baselines on both benchmarks' is an overstatement contradicted by Table 2 (MoCa-7B: 58.8 vs GME-7B: 59.3 on ViDoRe-v2) and Table 1 (MoCa-3B: 67.5 vs mmE5-11B: 69.8 on MMEB), but that is a correctness/accuracy concern, not circularity. No step in the paper reduces, by its own equations or by self-citation, to its own input.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical system, so its parameters are mostly tuned hyperparameters and its assumptions are domain-level transfer assumptions. No new physical entities or invented mechanisms are postulated.

free parameters (6)
  • MLM mask ratio = 0.4 (3B), 0.6 (7B)
    Chosen by validation in Section 3.5; affects the CPT objective.
  • MAE mask ratio = 0.5 (3B), 0.6 (7B)
    Chosen by validation in Section 3.5; affects the CPT objective.
  • MAE loss weight w = 0.5
    Balances MLM and MAE losses in the CPT objective; tuned on validation.
  • Contrastive temperature tau = 0.03
    Fixed in the contrastive loss (Eq. 3); not reported as tuned.
  • Number of hard negatives K = 2
    Each positive pair is accompanied by two hard negatives in Section 3.1.2.
  • Learning rates = 2e-6 (CPT), 1e-5 (CL)
    Reported in implementation details; chosen by empirical tuning.
assumptions (4)
  • standard math Cross-entropy and MSE losses are differentiable and optimize the model as written in Eq. (1) and (2).
    Standard loss functions used in MLM and MAE.
  • domain assumption Bidirectional attention is superior to causal attention for embedding tasks.
    Cited from text embedding literature [4,25,21,35], but Table 1 shows bidirectional alone is worse at 3B, so this only holds after CPT.
  • domain assumption Joint reconstruction on unlabeled interleaved text-image data transfers to downstream retrieval, classification, and grounding tasks.
    Central premise of Stage 1; supported only by benchmark gains, not by a mechanism analysis.
  • domain assumption Task-aware batching makes in-batch negatives harder and improves contrastive learning.
    Taken from [25] and ablated in Table 3; it is a training recipe not proven here beyond those ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings." pith.science (2026). https://pith.science/paper/DNKSUGZV

@misc{pith2026250623115,
  author       = {Pith},
  title        = {Pith review of: MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNKSUGZV}},
  note         = {Machine review of arXiv:2506.23115}
}
read the original abstract

Multimodal embedding models, built upon causal Vision Language Models (VLMs), have shown promise in various tasks. However, current approaches face three key limitations: the use of causal attention in VLM backbones is suboptimal for embedding tasks; scalability issues due to reliance on high-quality labeled paired data for contrastive learning; and limited diversity in training objectives and data. To address these issues, we propose MoCa, a two-stage framework for transforming pre-trained VLMs into effective bidirectional multimodal embedding models. The first stage, Modality-aware Continual Pre-training, introduces a joint reconstruction objective that simultaneously denoises interleaved text and image inputs, enhancing bidirectional context-aware reasoning. The second stage, Heterogeneous Contrastive Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple image-caption pairs to enhance generalization and alignment. Our method addresses the stated limitations by introducing bidirectional attention through continual pre-training, scaling effectively with massive unlabeled datasets via joint reconstruction objectives, and utilizing diverse multimodal data for enhanced representation robustness. Experiments demonstrate that MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results, and exhibits strong scalability with both model size and training data on MMEB.

Figures

Figures reproduced from arXiv: 2506.23115 by the authors.

Figure 1
Figure 1. Comparison of VLM-based multimodal embedding models. Left: Previous single￾stage contrastive learning with mainly image-caption pairs and causal attention. Right: MoCa. In modality-aware continual pre-training, we optimize a joint bidirectional reconstruction objective to denoise interleaved texts and images simultaneously. In heterogeneous contrastive fine-tuning, we train models to improve cross-modal fusion of th… view at source ↗
Figure 2
Figure 2. MoCa. (1) In modality-aware continual pre-training, the VLM backbone is trained to jointly reconstruct masked texts and images based on interleaved multimodal context with masked language modeling and masked autoencoding, respectively. (2) In heterogeneous contrastive fine-tuning, the VLM backbone from the previous stage is further fine-tuned with contrastive loss on a broad range of heterogeneous data. For each que… view at source ↗
Figure 3
Figure 3. Scaling effect of our CPT stage on downstream performance. We evaluate MMEB [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The performances of MoCa (3B) with different CPT settings on MMEB. MMEB using models trained with a fixed amount of data. We select hyperparameter values based on performance on validation sets, each containing 1K samples drawn from the corresponding training data. To …

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

    cs.CV 2026-08 conditional novelty 7.0 of 10

    UniME-R1 uses a failure-aware adviser to diagnose embedding mistakes from initial retrieval results and then either reranks candidates or re-retrieves with a feedback-based query rewrite.

  2. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

    cs.CV 2026-04 unverdicted novelty 6.5 of 10

    Using a structured rewrite instead of CoT as the generative interface improves MLLM-based multimodal embedding performance while cutting thinking tokens by about half.

  3. DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories

    cs.CV 2026-02 conditional novelty 6.5 of 10

    The paper reframes image retrieval as agentic exploration over personal visual histories and shows the best tested multimodal agent scores only 28.7 exact match on its new DISBench benchmark.

  4. Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

    cs.IR 2026-03 conditional novelty 5.0 of 10

    CoCoA forces an MLLM to reconstruct masked text through a single EOS token, improving multimodal embedding quality on MMEB-V1 and matching MoCa at 3B with far less pretraining data.

Reference graph

Works this paper leans on

58 extracted references · 16 canonical work pages · cited by 4 Pith papers

  1. [1]

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen ...

    arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    Qwen2.5-vl 10 technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl 10 technical rep...

    arXiv doi:10.48550/arxiv.2502.13923 2025

  3. [3]

    Vlmo: Unified vision- language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- language pre-training with mixture-of-modality-experts. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems 35: Annual Conf...

    2022

  4. [4]

    Llm2vec: Large language models are secretly powerful text encoders

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. CoRR, abs/2404.05961, 2024. doi: 10.48550/ARXIV .2404.05961. URL https: //doi.org/10.48550/arXiv.2404.05961

    arXiv doi:10.48550/arxiv.2404.05961 2024

  5. [5]

    mme5: Improving multimodal multilingual embeddings via high-quality synthetic data

    Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. CoRR, abs/2502.08468, 2025. doi: 10.48550/ARXIV .2502.08468. URL https://doi.org/ 10.48550/arXiv.2502.08468

    arXiv doi:10.48550/arxiv.2502.08468 2025

  6. [6]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. CoRR, abs/2501.17811, 2025. doi: 10.48550/ARXIV .2501.17811. URL https://doi.org/10.48550/arXiv.2501.17811

    arXiv doi:10.48550/arxiv.2501.17811 2025

  7. [7]

    UNITER: universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume ...

    doi:10.1007/978- 2020

  8. [8]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 2818–2829. IEEE...

    arXiv 2023

Show all 58 references
  1. [9]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of...

    2019

  2. [10]

    Colpali: Efficient document retrieval with vision language models

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28...

    2025

  3. [11]

    SEED-X: multimodal models with unified multi-granularity comprehension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. SEED-X: multimodal models with unified multi-granularity comprehension and generation. CoRR, abs/2404.14396, 2024. doi: 10.48550/ARXIV .2404.14396. URL https://doi.org/10.4855...

    2024 doi arXiv

  4. [12]

    Late chunking: Contextual chunk embeddings using long-context embedding models

    Michael Günther, Isabelle Mohr, Bo Wang, and Han Xiao. Late chunking: Contextual chunk embeddings using long-context embedding models. CoRR, abs/2409.04701, 2024. doi: 10. 48550/ARXIV .2409.04701. URLhttps://doi.org/10.48550/arXiv.2409.04701. 11

    2024 doi arXiv

  5. [13]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. CoRR, abs/2412.05237, 2024. doi: 10.48550/ARXIV .2412.05237. URL https://doi.org/10...

    2024 doi arXiv

  6. [14]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 15979– 15988. IE...

    2022

  7. [15]

    Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the ...

    2021

  8. [16]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2024

    2024 arXiv

  9. [17]

    Continual pre-training of language models

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview...

    2023

  10. [18]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vo...

    2021

  11. [19]

    Llave: Large language and vision embedding models with hardness-weighted contrastive learning

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness-weighted contrastive learning. CoRR, abs/2503.04812,

  12. [20]

    Building and better understanding vision-language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. CoRR, abs/2408.12637,

    arXiv

  13. [21]

    URL http://proceedings.mlr.press/v139/kim21k.html

  14. [22]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon ...

    2024

  15. [23]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, Intern...

    2022

  16. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edit...

    2023

  17. [25]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/2308.03281,

    arXiv

  18. [26]

    Nv-embed: Improved techniques for training llms as generalist embedding models

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore...

    2025

  19. [27]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  20. [28]

    Unify- ing multimodal retrieval via document screenshot embedding

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unify- ing multimodal retrieval via document screenshot embedding. In Yaser Al-Onaizan, Mo- hit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Languag...

    2024

  21. [29]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, Liang Zhao, Yisong Wang, Jiaying Liu, and Chong Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and ge...

    2024 doi arXiv

  22. [30]

    Introducing meta llama 3: The most capable openly available llm to date, April 2024

    Meta. Introducing meta llama 3: The most capable openly available llm to date, April 2024. https://ai.meta.com/blog/meta-llama-3/

    2024

  23. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In Marina Meila an...

    2021

  24. [32]

    Mm-embed: Universal multimodal retrieval with multimodal llms, 2024

    Sheng-chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms, 2024. URL https://arxiv.org/abs/2411.02571

    2024 arXiv

  25. [33]

    LAION-400M: open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114, 2021. URL https://arxiv.org/ab...

    2021 arXiv

  26. [34]

    From pixels to prose: A large dataset of dense image captions

    Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Gan- jdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels to prose: A large dataset of dense image captions. CoRR, abs/2406.10328, 2024. doi: 10.48550/ARXIV ...

    2024 doi arXiv

  27. [35]

    Repetition improves language model embeddings

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL h...

    2025

  28. [36]

    LXMERT: learning cross-modality encoder representations from transformers

    Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from transformers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- nati...

    2019

  29. [37]

    BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663, 2021. URL https://arxiv.org/abs/2104.08663

    2021 arXiv

  30. [38]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  31. [39]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533, 2022. doi: 10.48550/ARXIV .2212.03533. URL https://doi.org/ 10.48550/arXiv.2212.03533

    2022 doi arXiv

  32. [40]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s p...

    2024 doi arXiv

  33. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tie...

    2024 doi arXiv

  34. [42]

    Uniir: Training and benchmarking universal multimodal information retrievers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Co...

    2024 doi

  35. [43]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation. CoRR, abs/2410.13848, 2024. doi: 10.48550/ARXIV .2410.1...

    doi arXiv

  36. [44]

    C-pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors, Proceedings of the 14 47th Internatio...

    2024

  37. [45]

    Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J

    Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multi- lingual vision-lang...

    2025 doi arXiv

  38. [46]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. In The Thirteenth International Conference on Learning Representati...

    2025

  39. [47]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941–11952. IEEE, 2023. doi: 10.1109/ ICCV51070.2023.0...

    2023

  40. [48]

    Magiclens: Self-supervised image retrieval with open-ended instructions

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. O...

    2024

  41. [49]

    Gme: Improving universal multimodal retrieval by multimodal llms, 2024

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms, 2024. URL http://arxiv.org/abs/2412.16855

    2024 arXiv

  42. [50]

    QLIP: text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation

    Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. QLIP: text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. CoRR, abs/2502.05178, 2025. doi: 10.48550/ARXIV . 2502.051...

    2025 doi arXiv

  43. [51]

    Megapairs: Massive data synthesis for universal multimodal retrieval

    Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.14475, 2024. 15 Appendix A Detailed Results on MMEB We present the...

    2024 arXiv

  44. [52]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, ...

    2024

  45. [2019]

    URL https://doi.org/10.18653/v1/D19-1514

    doi: 10.18653/V1/D19-1514. URL https://doi.org/10.18653/v1/D19-1514

    doi

  46. [2021]

    URL http://proceedings.mlr.press/v139/jia21b.html

  47. [2023]

    URL https://doi.org/10.48550/arXiv.2308

    doi: 10.48550/ARXIV .2308.03281. URL https://doi.org/10.48550/arXiv.2308. 03281

    doi arXiv

  48. [2024]

    URL https://doi.org/10.48550/arXiv.2408

    doi: 10.48550/ARXIV .2408.12637. URL https://doi.org/10.48550/arXiv.2408. 12637

    doi arXiv

  49. [2025]

    URL https://doi.org/10.48550/arXiv.2503

    doi: 10.48550/ARXIV .2503.04812. URL https://doi.org/10.48550/arXiv.2503. 04812

    doi

  50. [4186]

    doi: 10.18653/V1/N19-1423

    Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1423. URL https://doi.org/10.18653/v1/n19-1423

    2019 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.