Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that reweighting InfoNCE gradients by a relative-similarity hardness score improves multi-modal embeddings, and reports a top MMEB score of 72.5 with the QQMM model.

desk verdict Tidy gradient analysis and a plausible hardness heuristic, but the SOTA claim is undercut by a numerical tie with UniME on the same backbone and an ablation that relies on an unreleased QQMM model. read the letter →

arxiv 2506.02020 v1 pith:7P3CY75E submitted 2025-05-28 cs.CV cs.LG

Youze Xue , Dian Li , Gang Liu This is my paper
classification cs.CVcs.LG
keywords multi-modalembeddingscontrastivelearningInfoNCEhardnegativemininggradientreweightingrelativesimilarityMLLMMMEB
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that in contrastive learning for multi-modal embeddings, a negative sample is hard not when its similarity to the query is absolutely high but when it is close to the positive sample's similarity. It introduces the Explicit Gradient Amplifier (EGA), a module that reweights the softmax probabilities in the InfoNCE gradient using a relative-similarity hardness score, so hard negatives exert a stronger push. With the LLaVA-OneVision-7B backbone, EGA matches or improves over prior MLLM embedding methods on the MMEB benchmark, and with the authors' QQMM model it reaches an average score of 72.5 and the top leaderboard position. A sympathetic reader would care because EGA is a drop-in, parameter-light change to the training objective that could transfer to any MLLM-based retrieval system.

What carries the argument

The central object is the Explicit Gradient Amplifier (EGA), a module that takes query and target embeddings, computes a hardness score for each negative given by $h_i^- = e^{\alpha(q\cdot x_i^- - q\cdot x^+)}$, multiplies the softmax probabilities $p_i^-$ by these scores, and renormalizes the products before using them to form the query and target gradients. The relative-similarity form of the hardness score carries the argument: a negative is marked as hard only when its similarity to the query is close to the positive's, so a moderately similar negative is easy when the positive is very similar and hard when the positive is weakly similar. EGA then injects the amplified probabilities into the InfoNCE gradient formulas, amplifying hard-negative contributions without changing the model architecture or the forward embedding computation.

What would settle it

Train the same LLaVA-OneVision-7B setup on MMEB with EGA at several values of $\alpha$ (for example 1, 5, 20, and 50) and on at least one other backbone; if the gains over the InfoNCE baseline disappear for $\alpha$ away from 20 or do not transfer across backbones, the claim that relative-similarity amplification is the mechanism is weakened. A sharper check is to search numerically for any loss function whose gradient equals the EGA gradient from Eq.3 and Eq.4; if none exists, EGA is a gradient heuristic, not a principled objective.

Watch

Extended reading notes

Core claim

The central claim is that the InfoNCE gradient with respect to the query decomposes into a weighted sum of differences between negative embeddings and the positive embedding, where the weight on each negative is the softmax probability $p_i^-$ that the query is classified as that negative. Therefore, the learning contribution of each negative is directly controlled by these probabilities, and amplifying the probabilities of hard negatives should yield more discriminative embeddings. EGA does this by computing a hardness score $h_i^- = e^{\alpha(q\cdot x_i^- - q\cdot x^+)}$ for each negative, multiplying each $p_i^-$ by its hardness score, renormalizing the products across negatives while preserving the total probability mass, and using the amplified probabilities in the gradient formulas. With the same LLaVA-OneVision-7B backbone, the paper reports state-of-the-art MMEB performance relative to VLM2Vec, LLaVE, and UniME, and with its QQMM backbone it reports an average score of 72.5 and the top rank on the MMEB leaderboard.

Load-bearing premise

The load-bearing premise is that replacing the true classification probabilities in the InfoNCE gradient with a sharpened, renormalized version, while leaving the positive gradient unchanged, is a valid optimization signal that truly improves embeddings rather than just distorting the gradient.

Editorial extensions

If this is right

  • On LLaVA-OneVision-7B, EGA matches or improves over VLM2Vec, LLaVE, and UniME on MMEB under the same training protocol, with particularly clear gains reported on out-of-distribution datasets.
  • With the QQMM backbone, the approach reports an average MMEB score of 72.5 and the top position on the leaderboard.
  • The ablations attribute roughly 1.8 points of average MMEB improvement to EGA itself and additional gains to the relative-similarity hardness score, particularly on retrieval and grounding tasks.
  • Because EGA is computed purely from embeddings with basic tensor operations, it adds negligible overhead and can be plugged into any MLLM-based embedding model without architectural changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the same gradient reweighting to unimodal contrastive learning, since the InfoNCE gradient analysis is modality-agnostic; if the mechanism is general, EGA should improve text-only or image-only retrieval benchmarks as well.
  • The top-rank result with QQMM is confounded by the backbone change, so an apples-to-apples comparison of EGA with other hard-negative methods on QQMM would isolate the module's contribution from the base model's strength.
  • One could check whether the optimal exponent $\alpha$ needs to anneal over training, since the hardness distribution changes as similarities sharpen; a fixed $\alpha=20$ may be near-optimal only for this particular backbone and batch protocol.
  • The paper does not show that the amplified gradient corresponds to the gradient of any actual loss function, so a useful follow-up would be to search for a loss whose gradient matches EGA; if none exists, the method should be understood as a gradient heuristic rather than a principled objective.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Explicit Gradient Amplifier (EGA), a method that modifies the gradient of the InfoNCE loss during contrastive training by reweighting the negative-sample probabilities with a hardness score h_i = exp(alpha * (q·x_i - q·x_+)), then substituting these reweighted values for the original softmax probabilities in the gradient computation. The authors argue that this explicitly amplifies the contribution of hard negatives and leads to more discriminative multi-modal embeddings. They evaluate on the MMEB benchmark using LLaVA-OneVision-7B and a proprietary QQMM model, reporting an overall score of 70.7 for LLaVA-OneVision and 72.5 for QQMM. A controlled ablation on QQMM shows an improvement from 70.4 baseline to 72.2 with EGA and 72.5 with the additional relative-similarity hardness (RS-H) score.

Significance. If the reported gains are reproducible, EGA is a simple, computationally lightweight, and potentially general technique for hard-negative exploitation in contrastive training of MLLM-based embedding models. The paper's strengths include a correct (albeit elementary) gradient derivation of InfoNCE, a plug-and-play design that is easy to implement, and evaluation on a public benchmark with released code. The principal weakness is the evidence base: the only controlled ablation uses a proprietary backbone whose details are not released, the same-backbone comparison with prior work is an exact tie rather than a state-of-the-art result, and no error bars or sensitivity analyses are provided. The central claim that EGA improves embedding learning therefore rests on a single, unreproducible run, and the theoretical analysis does not supply a substitute because the modified gradient is not the gradient of any stated loss.

major comments (4)
  1. [§4.2, Table 1] The abstract and Section 4.2 claim that the LLaVA-OneVision model 'achieves state-of-the-art performance' and shows 'particularly clear improvements on OOD', but Table 1 shows Ours (LLaVA-OneVision) averaging 70.7, exactly matching UniME (70.7), and being lower on GRD (90.4 vs 90.9) and OOD (65.6 vs 65.8). These claims are not supported by the table. Please either provide a controlled same-recipe comparison showing a real improvement or revise the claims to reflect parity with UniME.
  2. [§4.3, Table 2] The only controlled ablation of EGA is performed with QQMM, a proprietary model whose architecture and training details are not publicly described. The table reports no error bars, no seed variance, and no sensitivity analysis for alpha=20.0, so it is unclear whether the +1.8 and +0.3 point differences are statistically meaningful or reproducible. Add a controlled ablation on the publicly available LLaVA-OneVision backbone (baseline vs +EGA vs +EGA+RS-H) with multiple seeds, and report standard deviations and an alpha sweep.
  3. [§3.3, Eq. (7)] The text states that the reweighted values are 'normalized across all negative samples to ensure they form a valid probability distribution', but Eq. (7) does not produce a probability distribution. Specifically, sum_i \tilde{p}_i = (sum_i p_i) * (sum_i \hat{p}_i) / (sum_i \hat{p}_i) = sum_i p_i = 1 - p^+, which is less than 1 in general. Either correct the normalization so that the \tilde{p}_i sum to 1, or explicitly state that the total negative mass is intentionally preserved rather than re-normalized to a distribution.
  4. [§3.3, Eqs. (3)-(7)] The modified gradient is obtained by inserting reweighted probabilities into the InfoNCE gradient, but it is not shown to be the gradient of any loss function. Since the theoretical contribution of the paper is the gradient analysis, this gap makes the empirical evidence the decisive support for the method. Please either show that the modified gradient is a valid descent direction for a well-defined objective, or compare EGA against the equivalent loss-level reweighting (e.g., LLaVE's hardness weighting) to isolate the effect of gradient-level modulation.
minor comments (6)
  1. [Abstract and §4.2] The phrase 'consistently outperforms both VLM2Vec and LLaVE' is fine, but 'state-of-the-art' and 'particularly clear improvements on OOD' should be tempered because the table shows parity with UniME and a small OOD deficit.
  2. [§3.3, Eq. (5)] The hardness score uses dot products x·x_i and x·x_+, while the text refers to 'cosine similarity'. Please clarify whether the embeddings are L2-normalized so that dot products equal cosine similarities, and make the notation consistent.
  3. [§3.3, Step 2] The definition of the weighting matrix W as P - I requires a clear index convention. If P_{i,j} = \tilde{p}_j for row i, the diagonal terms are W_{i,i} = \tilde{p}_i - 1, which are negative and will push the positive pair together; the notation in the bullet list should be explicit about which index corresponds to the query and which to the target.
  4. [§4.1] The implementation details omit the optimizer, learning rate schedule, and warmup steps. These are needed for reproducibility, especially given that the only controlled ablation uses an unreleased model.
  5. [Figure 2] There is a typo in the caption: 'is integerated' should be 'is integrated'.
  6. [References] Reference [19] appears to have the author order garbled ('Shafiq Rayhan Joty Caiming Xiong Yingbo Zhou Semih Yavuz Rui Meng, Ye Liu' is not a standard citation format). Please check the metadata for this entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the gradient analysis is exact differentiation and EGA is an explicit design choice validated by ablation.

full rationale

The paper contains no load-bearing circular step. Section 3.2 derives the InfoNCE gradients in Eq.3 and Eq.4 by direct differentiation; this is a mathematical identity, not a claim that a prediction is derived from the method. Section 3.3 then defines EGA by explicitly reweighting the negative probabilities with the RS-H score in Eq.5-7; this is presented as a design choice ('we propose to explicitly amplify the gradients associated with hard negative samples'), and the paper does not claim that the modified probabilities are the gradient of any loss or use them as a derivation of a result. The central effectiveness claim rests on the controlled ablation in Table 2 (Baseline vs. +EGA vs. +EGA+RS-H), which is an empirical comparison rather than a derivation. The absence of a same-training-recipe LLaVA-OneVision baseline, the proprietary nature of QQMM, and the single choice of alpha=20.0 without sensitivity analysis are validity and reproducibility concerns, but they are not circularity under the stated criteria: no equation is shown to equal another by construction, and no fitted value is renamed as a prediction. No self-citation is load-bearing; the comparison rows are external published numbers, and the gradient decomposition itself is standard calculus. Therefore the derivation chain is self-contained and the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. It depends on standard InfoNCE assumptions and two hand-chosen hyperparameters. The core method is a heuristic reweighting with alpha tuned to the benchmark.

free parameters (2)
  • alpha = 20.0
    Rescales the hardness distribution in Eq.5; chosen by hand with no sensitivity analysis reported (Section 4.1).
  • temperature tau = 0.02
    InfoNCE temperature; standard hyperparameter but set without reported tuning (Section 4.1).
assumptions (4)
  • domain assumption InfoNCE loss with temperature tau is a valid training objective for MLLM embeddings.
    Used throughout Section 3.1; standard practice in VLM2Vec and LLaVE.
  • standard math The gradient decomposition in Eq.3 and Eq.4 is correct.
    It follows from differentiating Eq.1; not proven in the paper but mathematically straightforward.
  • ad hoc to paper Reweighting probabilities p_i^- by an exponential margin factor yields a better learning signal.
    Eq.5 to Eq.7 are introduced as a heuristic, not derived from any objective; the choice is validated only by ablation.
  • ad hoc to paper The relative-similarity hardness measure is more accurate than absolute similarity.
    Stated in Section 3.3; justified only by the 0.3 point improvement in Table 2 on one benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying." pith.science (2026). https://pith.science/paper/7P3CY75E

@misc{pith2026250602020,
  author       = {Pith},
  title        = {Pith review of: Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7P3CY75E}},
  note         = {Machine review of arXiv:2506.02020}
}
read the original abstract

With the rapid advancement of multi-modal large language models (MLLMs) in recent years, the foundational Contrastive Language-Image Pretraining (CLIP) framework has been successfully extended to MLLMs, enabling more powerful and universal multi-modal embeddings for a wide range of retrieval tasks. Despite these developments, the core contrastive learning paradigm remains largely unchanged from CLIP-style models to MLLMs. Within this framework, the effective mining of hard negative samples continues to be a critical factor for enhancing performance. Prior works have introduced both offline and online strategies for hard negative mining to improve the efficiency of contrastive learning. While these approaches have led to improved multi-modal embeddings, the specific contribution of each hard negative sample to the learning process has not been thoroughly investigated. In this work, we conduct a detailed analysis of the gradients of the info-NCE loss with respect to the query, positive, and negative samples, elucidating the role of hard negatives in updating model parameters. Building upon this analysis, we propose to explicitly amplify the gradients associated with hard negative samples, thereby encouraging the model to learn more discriminative embeddings. Our multi-modal embedding model, trained with the proposed Explicit Gradient Amplifier and based on the LLaVA-OneVision-7B architecture, achieves state-of-the-art performance on the MMEB benchmark compared to previous methods utilizing the same MLLM backbone. Furthermore, when integrated with our self-developed MLLM, QQMM, our approach attains the top rank on the MMEB leaderboard. Code and models are released on https://github.com/QQ-MM/QQMM-embed.

Figures

Figures reproduced from arXiv: 2506.02020 by the authors.

Figure 1
Figure 1. A template prompt for multi-modal inputs. A special <image> token is adopted to represent [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of the proposed multi-modal embedding model. The MLLMs [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An overview of the process for obtaining amplified gradients in the EGA module. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Illuminating Visual Identity in Universal Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.

  2. Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Visual token compression (4x fewer tokens) plus a three-stage generative/contrastive/judge-curated training pipeline yields state-of-the-art MLLM-based retrieval accuracy at lower inference cost.

  3. Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges

    cs.IR 2026-04 conditional novelty 4.0 of 10

    Qwen2.5-VL with CoT prompting reaches 82% zero-shot accuracy on geoscience document classification, beating the best multimodal embedding model (QQMM) at 63%.

Reference graph

Works this paper leans on

29 extracted references · 15 canonical work pages · cited by 3 Pith papers

  1. [1]

    Notellm: A retrievable large language model for note recommendation

    Chao Zhang, Shiwei Wu, Haoxin Zhang, Tong Xu, Yan Gao, Yao Hu, and Enhong Chen. Notellm: A retrievable large language model for note recommendation. In Companion Proceed- ings of the ACM Web Conference 2024, pages 170–179, 2024

    work page 2024

  2. [2]

    Notellm-2: Multimodal large representation models for recommendation

    Chao Zhang, Haoxin Zhang, Shiwei Wu, Di Wu, Tong Xu, Xiangyu Zhao, Yan Gao, Yao Hu, and Enhong Chen. Notellm-2: Multimodal large representation models for recommendation. arXiv preprint arXiv:2405.16789, 2024

    arXiv 2024

  3. [3]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024

    arXiv

  4. [4]

    VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

    Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Vdocrag: Retrieval-augmented generation over visually-rich documents. arXiv preprint arXiv:2504.09795, 2025

    work page Pith review arXiv 2025

  5. [5]

    Mmsearch: Unveiling the potential of large models as multi-modal search engines

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, Yu Liu, et al. Mmsearch: Unveiling the potential of large models as multi-modal search engines. In The Thirteenth International Conference on Learning Representations

    work page

  6. [6]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021

    work page 2021

  7. [7]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11941–11952. IEEE, 2023

    work page 2023

  8. [8]

    Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J

    Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense fea...

    arXiv 2025

Show all 29 references
  1. [9]

    EV A-CLIP-18B: scaling CLIP to 18 billion parameters.CoRR, abs/2402.04252, 2024

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. EV A-CLIP-18B: scaling CLIP to 18 billion parameters.CoRR, abs/2402.04252, 2024

    2024 arXiv

  2. [10]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  3. [11]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326, 2024

    2024 arXiv

  4. [12]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s p...

    2024 arXiv

  5. [13]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025 arXiv

  6. [14]

    Representation learning with contrastive predictive coding

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018

    2018 arXiv

  7. [15]

    Girshick

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726–9735. Computer Vision Foundation / IEEE, 2020

    2020

  8. [16]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR, 2020

    2020

  9. [17]

    Llave: Large language and vision embedding models with hardness-weighted contrastive learning

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness-weighted contrastive learning. CoRR, abs/2503.04812, 2025

    2025

  10. [18]

    Scaling deep contrastive learning batch size under memory limited setup

    Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. In RepL4NLP@ACL-IJCNLP, pages 316–321. Association for Computational Linguistics, 2021

    2021

  11. [19]

    Sfr- embedding-mistral:enhance text retrieval with transfer learning

    Shafiq Rayhan Joty Caiming Xiong Yingbo Zhou Semih Yavuz Rui Meng, Ye Liu. Sfr- embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024

    2024

  12. [20]

    Nv-embed: Improved techniques for training llms as generalist embedding models

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. In ICLR. OpenReview.net, 2025

    2025

  13. [21]

    Breaking the batch barrier (b3) of contrastive learning via smart batch mining

    Thirukovalluru Raghuveer, Rui Meng, Ye Liu, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, and Bhuwan Dhingra. Breaking the batch barrier (b3) of contrastive learning via smart batch mining. CoRR, abs/2505.11293, 2025

    2025

  14. [22]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. In ICLR. OpenRe- view.net, 2025

    2025

  15. [23]

    E5-V: universal embeddings with multimodal large language models

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: universal embeddings with multimodal large language models. CoRR, abs/2407.12580, 2024

    2024 arXiv

  16. [24]

    Breaking the modality barrier: Universal embedding learning with multimodal llms

    Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal embedding learning with multimodal llms. CoRR, abs/2504.17432, 2025. 9

    2025

  17. [25]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, pages 2818–2829. IEEE, 2023

    2023

  18. [26]

    Magiclens: Self-supervised image retrieval with open-ended instructions

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. In ICML. OpenReview.net, 2024

    2024

  19. [27]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR, 2023

    2023

  20. [28]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV (5), volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer, 2014

    2014

  21. [29]

    Visual news: Benchmark and challenges in news image captioning

    Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. In EMNLP (1), pages 6761–6771. Association for Computational Linguistics, 2021. 10

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.