Pith. sign in

REVIEW 3 major objections 6 minor 109 references

An MLLM embedder that throws away 75% of its visual tokens—through a fixed spatial shrink plus a three-stage training schedule and a judge-curated reranker—can retrieve better than full-token models, at a fraction of the latency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:16 UTC pith:DKDRIQKA

load-bearing objection Useful engineering with real speedups, but the central claim about compression being a strategic advantage is not actually tested. the 3 major comments →

arxiv 2602.05275 v2 pith:DKDRIQKA submitted 2026-02-05 cs.CV

Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

Qi Li , Yanzhe Zhao , Yongxin Zhou , Yameng Wang , Yandong Yang , Yuanjia Zhou , Jinxiang Liu This is my paper
classification cs.CV
keywords multimodal embeddingvisual token compressionMLLM-based retrievalcontrastive learninghard negative miningMLLM-as-a-Judgevisual document retrievaltoken efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to break the assumed trade-off between efficiency and accuracy in MLLM-based multimodal retrieval. Its claim is that the computational bottleneck—the long stream of redundant visual tokens that an MLLM must process—can be removed almost entirely: a fixed bilinear downsample of the visual feature map leaves only a quarter of the tokens, and this compression does not merely preserve performance but, with the right training, improves it. The authors build 2B and 8B models in this compressed architecture and train them in three stages: generative restoration, contrastive pretraining with hard-negative mining, and task-aware finetuning with an MLLM acting as judge. They report state-of-the-art results on the MMEB and VisDoc benchmarks, plus large gains on cross-modal retrieval, with 3–6× lower latency than uncompressed baselines. The paper's intended message is that the extra tokens were never the main source of retrieval signal; once the compressed visual stream is re-aligned and taught to discriminate, fewer tokens carry more.

Core claim

The paper's central claim is that visual token redundancy is not a necessary price for high-quality MLLM retrieval. By inserting a parameter-free bilinear interpolation between the vision encoder and the language model, the feature map is halved in each spatial dimension, reducing the visual token count to one quarter. The authors then show that this aggressively compressed model, when supported by a three-stage progressive training pipeline—first restoring generative multimodal ability, then contrastive pretraining with hard negatives, then task-aware finetuning on judge-verified data—outperforms its full-token counterparts and previous state-of-the-art systems on universal multimodal retri

What carries the argument

The load-bearing object is the parameter-free visual token compression module: bilinear interpolation Φ that downsamples the visual feature map F ∈ R^{H×W×C} to F′ ∈ R^{H/2×W/2×C} before the connector, cutting the token count from N to N/4. It is paired with a three-stage progressive training pipeline that does the heavy lifting: Stage 1 uses next-token prediction on a 32M-sample instruction corpus to re-align the compressed visual stream with the LLM; Stage 2 runs contrastive InfoNCE training on 16M samples, moving from in-batch warm-up to global hard-negative mining; Stage 3 finetunes on 1.5M judge-curated samples where an expert MLLM's logit comparison decides whether a retrieved candidat

Load-bearing premise

The claim stands or falls on whether a simple fixed shrink of the visual feature map to quarter size preserves the small details—text, numbers, tiny objects—that retrieval queries actually need; if that shrink discards those details, no training schedule can restore them.

What would settle it

Take the described 1.9B model and evaluate it on document queries that hinge on a single word, phone number, or small chart element, comparing against the original full-token backbone with identical settings. If fine-grained document retrieval collapses while coarse queries stay strong, the compression premise is wrong. A complementary test: evaluate with compression factor 3 or 4 (a 9× or 16× token cut) and watch whether a quality cliff appears; a smooth degradation would support the paper, an abrupt cliff would not.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval becomes deployable: the same or better accuracy with 75% fewer visual tokens means MLLM embedders can serve latency-critical, large-scale retrieval without specialized acceleration.
  • The training pipeline is the differentiator: compression alone hurts, but generative re-alignment plus hard-negative contrastive learning plus judge-curated finetuning recovers and exceeds full-token performance.
  • Smaller models become viable: a compressed 2B model can beat much larger uncompressed models, shifting the scaling economics for embedding services.
  • Two-stage retrieval benefits from compression: the efficient embedder feeds a reranker trained on the same judge-curated negatives, and the pair gains more than each component contributes separately.
  • Judge-curated data is a reusable asset: the same MLLM-judged sets that sharpen the embedder also train the reranker, so the curation cost is paid once and amortized.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the fixed compression factor s=2 is never ablated; sweeping s at evaluation time (1, 2, 3, 4) on the same checkpoint would reveal whether the reported gains hold under stronger compression or are specific to 25% token retention.
  • Beyond the paper: the 'parameter-free and universal' framing is cleaner for natural images than for documents, because the document results rely on a data-dependent tiling policy (up to 4 tiles) that itself performs additional spatial aggregation; a single-tile, fixed-resolution probe would separate the bilinear compression effect from tiling effects.
  • Beyond the paper: the MLLM-judge step converts relevance to a hard yes/no by comparing logits; using the logit margin as a soft negative weight, or tuning the threshold per dataset, is a low-cost avenue the paper does not explore.
  • Beyond the paper: the same compression-plus-restoration recipe should transfer to video or multi-frame retrieval, where token explosion is worse; the paper validates the idea on images and documents only.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Magic-MM-Embedding, a family of MLLM-based universal multimodal embedding models that reduce visual token count by 75% via a parameter-free bilinear interpolation downsampling module (InternVL3-VTC). The method is trained with a three-stage progressive pipeline: (1) generative restoration of multimodal understanding on 32M instruction samples, (2) contrastive pretraining on 16M retrieval samples with global hard negative mining, and (3) task-aware finetuning on 1.5M judge-curated samples using Qwen3-VL as an MLLM judge, followed by a synergistic reranker. The authors report state-of-the-art results on MMEB (70.2/72.8 average with E/E+R for 2B/8B) and VisDoc (73.3/75.8), with substantially lower inference latency than prior MLLM embedders.

Significance. If the central causal claim were rigorously established, the paper would make a practically important contribution: showing that aggressive visual token compression can be a strategic advantage rather than a trade-off for MLLM-based retrieval, with 3-6x latency reductions at state-of-the-art accuracy. The paper is commendable for its clear experimental design, detailed reporting of training data and hyperparameters, and internally consistent benchmark results. However, the headline claim that compression 'significantly outperform[s] its non-compressed counterparts' is not supported by the only controlled comparison (Table 9), which shows near-parity. The paper also lacks ablations on the compression factor and error bars, and selects multiple hyperparameters on the same test sets used for the headline numbers. These gaps prevent the reader from attributing the observed gains to the compression module versus the large-scale training data, judge-based curation, or reranker.

major comments (3)
  1. [§5.4, Table 9] The only controlled comparison between vanilla InternVL3 and InternVL3-VTC, trained identically on the Stage-2 warm-up, shows essentially parity: MMEB 62.9 vs 63.7, VisDoc 68.4 vs 68.5. This directly contradicts the claim in §1 that 'a model with aggressive visual token compression can significantly outperform its non-compressed counterparts.' The headline SOTA numbers in Tables 3–5 compare against baselines that differ in backbone, data scale (16M stage-2 + 32M stage-1 + 1.5M stage-3), judge curation, and reranker, so those gains cannot be attributed to the compression module or the co-designed pipeline. Please provide a matched control: train vanilla InternVL3 through the full three-stage pipeline (or at least stages 2+3 with identical data and judge curation) and compare. If this is infeasible, the central claim must be substantially softened.
  2. [§3.2, Eq. (3)] The compression factor s is never varied in any ablation. The method's core assumption is that bilinear downsampling by 2× preserves retrieval-relevant information, including fine-grained text details in visual documents, after Stage-1 generative re-alignment. Without ablations over s (e.g., s=1, 2, 4) or alternative interpolation kernels, the reader cannot determine whether s=2 is load-bearing or whether the performance is driven entirely by the training pipeline. This is particularly important for VisDoc, where the paper's own tiling policy (MAX_NUM=4 for documents, §5.2) complicates the 'parameter-free and universal' claim. Please add these ablations.
  3. [§5.4, Tables 7, 8, 10] Multiple hyperparameters—the number of hard negatives n (Table 8), LoRA rank (Table 10), and the pipeline components themselves (Table 7)—are selected on the same MMEB and VisDoc evaluation sets used for the headline results. No error bars or variance estimates are reported. This selection-on-test-set practice can inflate reported gains and makes the statistical significance of the improvements unclear. Please report results with multiple seeds or use a held-out validation split for model selection, and provide variance for the main results in Tables 3–5.
minor comments (6)
  1. [Abstract and §1] The phrasing 'significantly outperform its non-compressed counterparts' is not supported by Table 9. Please rephrase to reflect the actual evidence, or present the matched-control results.
  2. [§5.2] The data-dependent tiling policy (MAX_NUM=4 for visual documents, 1 otherwise) is a task-specific design choice. While it improves document retrieval, it weakens the 'universal' and 'parameter-free' claims. Please discuss this explicitly as a hyperparameter and consider ablating MAX_NUM.
  3. [Table 6] The latency comparison includes vanilla InternVL3 as a baseline, but the corresponding performance numbers are not shown in that table. Please cross-reference Table 9 to make clear that the parity in accuracy accompanies the latency gain.
  4. [§3.3] Stage 2 'Global Hard Negative Mining' samples only 2 hard negatives per query from positions 50–100. The term 'global' is somewhat misleading—it refers to per-sub-dataset mining, not a global corpus. Please clarify.
  5. [General] The paper has several typos and grammatical errors, e.g., 'constrastive' in §3.3, 'traing' in §5.2, 'presents' in §5.3, 'comparsion' in §5.3. A careful proofread is needed.
  6. [§3.2] The compression module itself is parameter-free, but the full model includes LoRA adapters and a reranker. The text should distinguish between the compression module being parameter-free and the overall system having learned parameters.

Circularity Check

0 steps flagged

No significant circularity: benchmark claims are empirical, the compression module is independently defined, and the MLLM judge is external; the missing controlled comparison is a support gap, not circularity.

full rationale

The paper's central claims are empirical results on MMEB, VisDoc, and cross-modal benchmarks against external baselines. The visual token compression is defined by Eq. 3 (bilinear interpolation with H'=H/s, W'=W/s), which is an architectural preprocessing step independent of the evaluated metrics; no equation is fit to the benchmark outcomes. The three-stage pipeline is a training scheme, not a derivation that assumes its conclusion. The MLLM-as-a-Judge in Stage 3 is Qwen3-VL [1], an independently pretrained model, so its curated hard negatives are external supervision rather than a self-referential uniqueness or ansatz argument. The reranker is initialized from the Stage-1 checkpoint and trained on judge-curated data; this creates a self-training loop in data generation, but it does not make the headline comparisons reduce to the paper's own outputs by construction. No load-bearing self-citation appears: the only apparent author self-citation is ref [57] (Jinxiang Liu et al.) on sound localization, which is not used to justify any central premise. The main limitation—that the controlled compression ablation (Table 9) shows only parity (MMEB 63.7 vs 62.9; VisDoc 68.5 vs 68.4) and that no vanilla InternVL3 baseline is run through the full three-stage pipeline—undermines the causal attribution of gains to compression, but under-support is not circularity. Therefore no circular step can be exhibited, and the score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper's contribution is entirely empirical; it introduces no new theoretical entities. The main free parameters are training and architecture choices, several of which were selected by evaluating on the same MMEB/VisDoc aggregates that appear in the headline tables. The axioms are standard domain assumptions for this line of work, with the strongest being the preservation of fine-grained visual information under aggressive bilinear compression.

free parameters (6)
  • Compression factor s = 2 (75% token reduction)
    Chosen by hand; no ablation sweep; central architectural knob (§3.2).
  • Number of hard negatives n = 12 for stage 3, 2 for stage 2 Global-HNM
    Selected via ablation on MMEB/VisDoc test averages (Table 8); a tuning-on-eval parameter.
  • LoRA rank = 16
    Selected via ablation on MMEB/VisDoc averages (Table 10).
  • Temperature tau = 0.03
    Set a priori, standard for InfoNCE; not ablated.
  • MAX_NUM image tiles = 4 for documents, 1 for natural images
    Data-dependent policy; chosen for efficiency and to preserve document detail (§5.2).
  • Hard negative sampling positions = positions 50-100
    Heuristic to avoid false negatives in top-10; not ablated (§3.3).
axioms (5)
  • domain assumption Bilinear interpolation preserves sufficient semantic and fine-grained visual information for retrieval after re-alignment training
    §3.2 and §5.3; no analytical guarantee; empirically argued.
  • domain assumption Qwen3-VL-7B logit(yes)>logit(no) provides reliable relevance labels for hard-negative curation
    §3.3 Stage 3; judge trust is assumed; errors would inject false labels.
  • domain assumption InfoNCE with a single [EOS] embedding is a sufficient objective for universal multimodal retrieval
    §3.1 Eq. (2); inherited from VLM2Vec/UniME line.
  • domain assumption MEBE-train/MMEB-train and other public corpora contain enough signal to train a universal embedder
    §4 data construction; scale hypothesis.
  • domain assumption Existing baselines were compared fairly with matched resolutions and protocol
    §5.1; relies on the authors' re-implementation of inference settings.

pith-pipeline@v1.3.0-alltime-deepseek · 24117 in / 12379 out tokens · 122496 ms · 2026-08-03T04:16:57.346058+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have shown immense promise in universal multimodal retrieval, which aims to find relevant items of various modalities for a given query. However, their practical application is often hindered by the substantial computational cost incurred from processing a large number of tokens from visual inputs. In this paper, we propose Magic-MM-Embedding, a series of novel models that achieve both high efficiency and state-of-the-art performance in universal multimodal embedding. Our approach is built on two synergistic pillars: (1) a highly efficient MLLM architecture incorporating visual token compression to drastically reduce inference latency and training time, and (2) a multi-stage progressive training strategy designed to not only recover but significantly boost performance. This coarse-to-fine training paradigm begins with extensive continued training to restore multimodal understanding and generation capabilities, progresses to large-scale contrastive pretraining and hard negative mining to enhance discriminative power, and culminates in a task-aware fine-tuning stage guided by an MLLM-as-a-Judge for precise data curation. Comprehensive experiments show that our model outperforms existing methods by a large margin while being more inference-efficient.

Figures

Figures reproduced from arXiv: 2602.05275 by Jinxiang Liu, Qi Li, Yameng Wang, Yandong Yang, Yanzhe Zhao, Yongxin Zhou, Yuanjia Zhou.

Figure 1
Figure 1. Figure 1: Breaking the efficiency-performance trade-off for MLLM embedders for universal multimodal retrieval. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed visual-token-efficient architecture for universal multimodal retrieval. (a) The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

109 extracted references · 30 linked inside Pith

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Coig-cqia: Quality is all you need for chinese instruction fine-tuning, 2024

    Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al. Coig-cqia: Quality is all you need for chinese instruction fine-tuning, 2024

    2024

  3. [3]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.Advances in neural information processing systems, 32, 2019

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.Advances in neural information processing systems, 32, 2019

    2019

  4. [4]

    Baai-mtp dataset

    Beijing Academy of Artificial Intelligence. Baai-mtp dataset. https://data.baai.ac.cn/datadetail/BAAI-MTP. Accessed: 2026-01-24

    2026

  5. [5]

    Flame: Frozen large language models enable data-efficient language-image pre-training.arXiv:2411.11927, 2024

    Anjia Cao, Xing Wei, and Zhiheng Ma. Flame: Frozen large language models enable data-efficient language-image pre-training.arXiv:2411.11927, 2024. 12

    Pith/arXiv arXiv 2024

  6. [6]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024

    2024

  7. [7]

    Webqa: Multihop and multimodal qa

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495–16504, 2022

    2022

  8. [8]

    mme5: Improving multimodal multilingual embeddings via high-quality synthetic data.arXiv preprint arXiv:2502.08468, 2025

    Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data.arXiv preprint arXiv:2502.08468, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  10. [10]

    Reproducible scaling laws for contrastive language-image learning.arXiv:2212.07143, 2022

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning.arXiv:2212.07143, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Think then embed: Generative context improves multimodal embedding.arXiv preprint arXiv:2510.05014, 2025

    Xuanming Cui, Jianpeng Cheng, Hong-you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xichen Pan, Chaitanya Ahuja, Shlok Kumar Mishra, Yonghuan Yang, Jun Xiao, et al. Think then embed: Generative context improves multimodal embedding.arXiv preprint arXiv:2510.05014, 2025

    arXiv 2025

  12. [12]

    Visual dialog

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 326–335, 2017

    2017

  13. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  14. [14]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  15. [15]

    Pact: Pruning and clustering-based token reduction for faster visual language models

    Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, and Aymen Shabou. Pact: Pruning and clustering-based token reduction for faster visual language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 14582–14592, 2025

    2025

  16. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, ICLR 2021, V...

    2021

  17. [17]

    The pascal visual object classes challenge: A retrospective.International journal of computer vision, 111(1):98–136, 2015

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective.International journal of computer vision, 111(1):98–136, 2015

    2015

  18. [18]

    Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449, 2024

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344, 2023

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Vl-clip: Enhancing multimodal recommendations via visual grounding and llm-augmented clip embeddings

    Ramin Giahi, Kehui Yao, Sriram Kollipara, Kai Zhao, Vahid Mirjalili, Jianpeng Xu, Topojoy Biswas, Evren Korpeoglu, and Kannan Achan. Vl-clip: Enhancing multimodal recommendations via visual grounding and llm-augmented clip embeddings. InProceedings of the Nineteenth ACM Conference on Recommender Systems, pages 482–491, 2025. 13

    2025

  21. [21]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data.arXiv preprint arXiv:2410.18558, 2024

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data.arXiv preprint arXiv:2410.18558, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Breaking the modality barrier: Universal embedding learning with multimodal llms

    Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal embedding learning with multimodal llms. InProceedings of the 33rd ACM International Conference on Multimedia, MM ’25, page 2860–2869, New York, NY , USA, 2025. Association for Computing...

    2025

  23. [23]

    Unime-v2: Mllm-as-a-judge for universal multimodal embedding learning.AAAI, 2026

    Tiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang, Tom Weidong Cai, Jiankang Deng, and Lidong Bing. Unime-v2: Mllm-as-a-judge for universal multimodal embedding learning.AAAI, 2026

    2026

  24. [24]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018

    2018

  25. [25]

    Efficient multimodal learning from data-centric perspective.arXiv preprint arXiv:2402.11530, 2024

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective.arXiv preprint arXiv:2402.11530, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    The many faces of robustness: A critical analysis of out-of- distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of- distribution generalization. InProceedings of the IEEE/CVF international conference on computer vision, pages 8340–8349, 2021

    2021

  27. [27]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262–15271, 2021

    2021

  28. [28]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36:31096–31116, 2023

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36:31096–31116, 2023

    2023

  29. [29]

    Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities

    Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12065– 12075, 2023

    2023

  30. [30]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  31. [31]

    Videorag: Retrieval-augmented generation over video corpus.arXiv preprint arXiv:2501.05874, 2025

    Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrieval-augmented generation over video corpus.arXiv preprint arXiv:2501.05874, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. PMLR, 2021

    2021

  33. [33]

    Rzenembed: Towards comprehensive multimodal retrieval.CoRR, abs/2510.27350, 2025

    Weijian Jian, Yajun Zhang, Dawei Liang, Chunyu Xie, Yixiao He, Dawei Leng, and Yuhui Yin. Rzenembed: Towards comprehensive multimodal retrieval.CoRR, abs/2510.27350, 2025

    arXiv 2025

  34. [34]

    E5-v: Universal embeddings with multimodal large language models

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, De- qing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv:2407.12580, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks.ICLR, 2025

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks.ICLR, 2025

    2025

  36. [36]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 14

    2014

  37. [37]

    The hateful memes challenge: Detecting hate speech in multimodal memes.Advances in neural information processing systems, 33:2611–2624, 2020

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes.Advances in neural information processing systems, 33:2611–2624, 2020

    2020

  38. [38]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123(1):32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123(1):32–73, 2017

    2017

  39. [39]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.International journal of computer vision, 128(7):1956–1981, 2020

    1956

  40. [40]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  41. [41]

    Llave: Large language and vision embedding models with hardness-weighted contrastive learning.CoRR, abs/2503.04812, 2025

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness-weighted contrastive learning.CoRR, abs/2503.04812, 2025

    arXiv 2025

  42. [42]

    Ume-r1: Exploring reasoning-driven generative multimodal embeddings.arXiv preprint arXiv:2511.00405, 2025

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Ume-r1: Exploring reasoning-driven generative multimodal embeddings.arXiv preprint arXiv:2511.00405, 2025

    arXiv 2025

  43. [43]

    Building and better understanding vision-language models: insights and future directions.arXiv preprint arXiv:2408.12637, 2024

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions.arXiv preprint arXiv:2408.12637, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Infinity instruct: Scaling instruction selection and synthesis to enhance language models.arXiv preprint arXiv:2506.11116, 2025

    Jijie Li, Li Du, Hanyu Zhao, Bo-wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis to enhance language models.arXiv preprint arXiv:2506.11116, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  47. [47]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  48. [48]

    Silkie: Preference distillation for large visual language models.arXiv preprint arXiv:2312.10665, 2023

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models.arXiv preprint arXiv:2312.10665, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking.arXiv preprint arXiv:2601.04720, 2026

    Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking.arXiv preprint arXiv:2601.04720, 2026

    Pith/arXiv arXiv 2026

  50. [50]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26763–26773, 2024

    2024

  51. [51]

    Mm- embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm- embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014. 15

    2014

  53. [53]

    Gres: Generalized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592–23601, 2023

    2023

  54. [54]

    Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Visual news: Benchmark and challenges in news image captioning

    Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 6761–6771, 2021

    2021

  56. [56]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024

    2024

  57. [57]

    Exploiting transformation invariance and equivariance for self-supervised sound localisation

    Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. InProceedings of the 30th ACM International Conference on Multimedia, pages 3742–3753, 2022

    2022

  58. [58]

    Edis: Entity-driven image search over multimodal web content

    Siqi Liu, Weixi Feng, Tsu-Jui Fu, Wenhu Chen, and William Wang. Edis: Entity-driven image search over multimodal web content. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4877–4894, 2023

    2023

  59. [59]

    Lamra: Large multimodal model as your advanced retrieval assistant.CVPR, 2024

    Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant.CVPR, 2024

    2024

  60. [60]

    Image retrieval on real-life images with pre-trained vision-and-language models

    Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. InProceedings of the IEEE/CVF international conference on computer vision, pages 2125–2134, 2021

    2021

  61. [61]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  62. [62]

    Unifying multimodal retrieval via document screenshot embedding.arXiv preprint arXiv:2406.11251, 2024

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding.arXiv preprint arXiv:2406.11251, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    Visa: Retrieval augmented generation with visual source attribution

    Xueguang Ma, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Wenhu Chen, and Jimmy Lin. Visa: Retrieval augmented generation with visual source attribution. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 30154–30169, 2025

    2025

  64. [64]

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.Advances in Neural Information Processing Systems, 37:95963–96010, 2024

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.Advances in Neural Information Processing Systems, 37:95963–96010, 2024

    2024

  65. [65]

    Vidore benchmark v2: Raising the bar for visual retrieval.arXiv preprint arXiv:2505.17166, 2025

    Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for visual retrieval.arXiv preprint arXiv:2505.17166, 2025

    arXiv 2025

  66. [66]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016

    2016

  67. [67]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  68. [68]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022

    2022

  69. [69]

    In- fographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. In- fographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. 16

    2022

  70. [70]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  71. [71]

    Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents.arXiv preprint arXiv:2507.04590, 2025

    Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, et al. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents.arXiv preprint arXiv:2507.04590, 2025

    Pith/arXiv arXiv 2025

  72. [72]

    Ops-mm-embedding-v1

    OpenSearch-AI. Ops-mm-embedding-v1. https://huggingface.co/OpenSearch-AI/ Ops-MM-embedding-v1-2B. Accessed: 2026-01-24

    2026

  73. [73]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015

    2015

  74. [74]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine...

    2021

  75. [75]

    A- okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. InEuropean conference on computer vision, pages 146–162. Springer, 2022

    2022

  76. [76]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. InProceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019

    2019

  77. [77]

    Sharegpt-chinese-english-90k: A bilingual chinese-english human-machine dialogue dataset

    ShareAI Lab. Sharegpt-chinese-english-90k: A bilingual chinese-english human-machine dialogue dataset. https://huggingface.co/datasets/shareAI/ShareGPT-Chinese-English-90k, 2023. Hugging Face dataset repository

    2023

  78. [78]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  79. [79]

    Eva-clip-18b: Scaling clip to 18 billion parameters.arXiv:2402.04252, 2023

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip to 18 billion parameters.arXiv:2402.04252, 2023

    Pith/arXiv arXiv 2023

  80. [80]

    Breaking the batch barrier (b3) of contrastive learning via smart batch mining

    Raghuveer Thirukovalluru, Rui Meng, Ye Liu, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, et al. Breaking the batch barrier (b3) of contrastive learning via smart batch mining. arXiv preprint arXiv:2505.11293, 2025

    arXiv 2025

Showing first 80 references.