Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single 865M-parameter embedding model can retrieve text, images, and visually rich documents across 30 languages, and its vectors can be cut to 256 dimensions with under 1% quality loss.

desk verdict Useful multilingual dual encoder with a solid text/crossmodal core, but the headline ViDoRe gain is unverified due to likely train/test leakage. read the letter →

arxiv 2412.08802 v2 pith:ROMSPYG2 submitted 2024-12-11 cs.CL cs.CVcs.IR

Andreas Koukounas , Georgios Mastrapas , Sedigheh Eslami , Bo Wang , Mohammad Kalim Akram , Michael Günther , Isabelle Mohr , Saba Sturua
show 2 more authors
Nan Wang Han Xiao
This is my paper · ORCID
classification cs.CLcs.CVcs.IR
keywords multilingualembeddingscontrastivelearningcrossmodalretrievalvisualdocumentMatryoshkarepresentationzero-shottextvision-languagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that one open-weight embedding model can cover text-only retrieval, semantic similarity, image-to-text and text-to-image retrieval, and visually rich document retrieval across many languages. Its central move is to extend the jina-clip-v1 recipe, which jointly optimizes text-text and text-image contrastive objectives, with a multilingual text encoder, escalating image resolution, and Matryoshka Representation Learning. The authors report that jina-clip-v2 outperforms previous CLIP-style models on multilingual crossmodal retrieval, text retrieval, and STS, and lands close to the dedicated text embedder jina-embeddings-v3 on text-only benchmarks. If true, the practical consequence is that one model can replace several separate embedding systems, and its vectors can be truncated to a quarter of their size with negligible quality loss.

What carries the argument

The load-bearing mechanism is a multi-task, multi-stage contrastive training loop. At every stage the model minimizes an InfoNCE loss, a contrastive objective that pulls paired query-target embeddings together and pushes in-batch negatives apart, and stage 3 adds a hard-negative extension that pits each query against seven explicit negatives. The text encoder is initialized from the multilingual Jina-XLM-RoBERTa checkpoint and the image encoder from EVA02-L/14, and the stages progressively raise text context length from 77 to 512 tokens and image resolution from 224 to 512 pixels. In addition, every loss is recomputed on truncated embeddings of 64, 128, 256, 512, and 768 dimensions, which is what trains the model so that smaller output vectors remain useful.

What would settle it

Retrain jina-clip-v2 without the ViDoRe-derived synthetic data and re-evaluate on the ViDoRe test tasks; if the reported 35% improvement over jina-clip-v1 collapses when that split is removed, the gain is partly a benchmark-leakage artifact rather than a genuine visual-document capability.

Watch

Extended reading notes

Core claim

The paper's central claim is that a dual-encoder model, jina-clip-v2, trained with a multi-task, multi-stage contrastive objective, can act as both a multilingual text embedder and a multilingual vision-language embedder without sacrificing either role. The authors report that it outperforms prior CLIP-style models on zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval in English and across 29 non-English languages, and that on the MTEB retrieval and STS suites it lands close to the dedicated multilingual text embedder jina-embeddings-v3. They further claim that the model understands visually rich documents such as tables, graphs, infographics, and PDF-style pages far better than jina-clip-v1, reaching 52.65 average nDCG@5 on the ViDoRe benchmark, and that its 1,024-dimension embeddings can be truncated to 256 dimensions with under 1% loss thanks to Matryoshka Representation Learning.

Load-bearing premise

The visual-document retrieval results assume that the ViDoRe synthetic training examples included in the multimodal datasets do not overlap the ViDoRe benchmark's test queries and documents, because the paper does not state that those test items were excluded during pretraining.

Editorial extensions

If this is right

  • One deployment can serve English and multilingual text retrieval, STS, and both directions of image-text retrieval, replacing separate text and vision embedding models.
  • Embeddings truncated from 1,024 to 256 dimensions retain over 99% of full-size performance, cutting storage and bandwidth costs substantially.
  • Visual document retrieval over tables, infographics, scientific graphs, and PDF-style pages becomes practical with a CLIP-style dual encoder, reaching 52.65 average nDCG@5 on ViDoRe.
  • Multilingual gains over the English-only predecessor are large: up to 67% higher crossmodal scores, up to 60% higher text retrieval scores, and up to 43% higher STS scores on the reported benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ViDoRe synthetic training examples overlap the ViDoRe benchmark's test documents, part of the reported 35% visual-document improvement may reflect benchmark leakage rather than a new capability; checking the split provenance would settle this.
  • The slight crossmodal drop from stage 2 to stage 3, while text retrieval improves, hints that a single shared embedding space has a ceiling for jointly optimizing both objectives, so task-specific adapters or separate heads may be a natural next step.
  • The 256-dimension truncation claim is demonstrated only on the benchmarks in this paper; a harder test would be truncating embeddings during out-of-distribution retrieval on new languages and document types.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces jina-clip-v2, an 865M-parameter dual-encoder model that combines a multilingual Jina-XLM-RoBERTa text encoder and an EVA02-L/14 vision encoder. Training uses a multi-task, multi-stage contrastive objective with InfoNCE losses on text pairs, text triplets with hard negatives, and image-text pairs, including multilingual data and visually rich document images. The model is evaluated on English and multilingual crossmodal retrieval, MTEB retrieval and STS tasks, the ViDoRe visual document retrieval benchmark, and an ablation of Matryoshka Representation Learning. The central claims are that jina-clip-v2 outperforms prior CLIP-based models on multilingual crossmodal and text-only tasks, achieves a 35% improvement over jina-clip-v1 on ViDoRe, and retains performance when embeddings are truncated to 256 dimensions.

Significance. If the reported results are trustworthy, jina-clip-v2 is a practically valuable open-weight model: it provides a single encoder for text-only, crossmodal, and visually rich document retrieval across many languages, with transparent training-stage details, public model weights, and a Matryoshka-style dimensionality trade-off that loses under 1% performance at 256 dimensions. The explicit multi-stage training recipe and the analysis of resolution effects are useful additions to the CLIP literature. The main significance risk is that the headline ViDoRe result rests on a train/test provenance gap, and the MTEB text-only comparison has a related provenance question; both need to be resolved before the central claims can be taken at face value.

major comments (3)
  1. [§3.1 and §4.3] The ViDoRe benchmark result is load-bearing for the visual document retrieval contribution, but the paper does not establish that the benchmark test set is disjoint from the training data. Section 3.1 states that Dmm;s and Dmm;l include 'ViDoRe synthetic training data (Faysse et al., 2024)', and Section 4.3 reports an average nDCG@5 of 52.65% and a 35% improvement over jina-clip-v1 on the ViDoRe benchmark. Because the ViDoRe synthetic training data is derived from the same benchmark project, and the paper gives no statement of excluded queries or documents or of deduplication, the reported gain could partly reflect train/test leakage. Please provide a precise description of the split between the synthetic training data and the benchmark test collections, report any deduplication performed, and if overlap exists, re-evaluate without the contaminated training data.
  2. [§5.1] The resolution analysis in Section 5.1 is affected by the same provenance issue. The four additional runs are trained 'using the same visually rich training set' and then evaluated on the ViDoRe benchmark; since that training set includes ViDoRe synthetic data, the nDCG@5 curve in Figure 2 may partially measure the model's ability to memorize training-page images rather than the effect of resolution on generalization. Please rerun the resolution check on a training set that provably excludes ViDoRe benchmark documents, or explicitly argue why overlap cannot affect the relative ranking across resolutions.
  3. [§3.1 and §4.2] The text-retrieval evaluation has a parallel provenance gap. Dtxt;p and Dtxt;t are described as 'data in 30 languages ... introduced as training data for jina-embeddings-v3', and Section 4.2 evaluates on MTEB retrieval and STS tasks. Since many MTEB retrieval datasets (e.g., MS MARCO, CQADupstack, FEVER) are standard training sources for text embedding models, the paper should state whether any MTEB test split was included in Dtxt;p or Dtxt;t, and if so, report results on the held-out subsets. Without this, the comparison against jina-embeddings-v3 on retrieval and STS is hard to interpret.
minor comments (5)
  1. [§4.3] The phrase 'toping jina-clip-v1' should be corrected to 'topping jina-clip-v1'.
  2. [§1, contribution list] The sentence 'compared jina-clip-v1' is missing the word 'to'; it should read 'compared to jina-clip-v1'.
  3. [Table 1] Table 1 states '89 languages supported' for the text encoder while the abstract and Section 3.1 refer to 30 languages in the training data; please clarify whether 89 refers to the underlying XLM-RoBERTa vocabulary support and 30 to the actual training languages, and state this explicitly in the text.
  4. [Equation (1)] The summation notation 'kP' appears to be a rendering error for the sum over k items; please fix the typesetting so the denominator is unambiguous.
  5. [References] The NLLB-CLIP technical report is cited twice as Visheratin (2023a) and Visheratin (2023b); these should be consolidated into a single reference entry.

Circularity Check

1 steps flagged · score 6.0 of 10

The ViDoRe visual-document-retrieval result is trained on the same benchmark family, and the paper never documents a disjoint split, so the reported 52.65 nDCG@5 and 35% gain over jina-clip-v1 are at least partly a fit to the evaluation input rather than an independent prediction.

  1. other [Section 3.1 (training data) and Section 4.3 (ViDoRe benchmark evaluation, Table 4)]
    "Specifically, the short caption image-text dataset Dmm;s and the long-caption Dmm;l include the following training datasets: DocVQA (Mathew et al., 2021b), TatDQA (Zhu et al., 2022), InfographicsVQA (Mathew et al., 2021a), SciGraphQA (Li & Tajbakhsh, 2023), ArXivQA and ArXivCAP (Li et al., 2024), WIT (Srinivasan et al., 2021) and ViDoRe synthetic training data (Faysse et al., 2024)."

    Section 4.3 reports jina-clip-v2 as outperforming other CLIP models on the ViDoRe Benchmark with average nDCG@5 of 52.65%, while Section 3.1 says the training mixture includes ViDoRe synthetic training data and the same QA/visual-document datasets (DocVQA, TatDQA, InfographicsVQA, SciGraphQA, ArXivQA) that constitute ViDoRe. The paper never states that the benchmark's test queries and pages were excluded from these training sources. For QA sources, the paper says training text is built by concatenating 'the query and answer for each sample'; if those samples include benchmark queries/pages, the ViDoRe score partly measures memorization of the training distribution. The benchmark score is thus at least partially an input presented as an output.

full rationale

The rest of the paper is not circular in the equation-level sense. The crossmodal, MTEB, and Matryoshka evaluations use external benchmarks (CLIP Benchmark, Crossmodal-3600, XTD10, MTEB), and the MRL truncation result is a designed training property, not a hidden input. The many self-citations (Koukounas et al. 2024, Sturua et al. 2024) are used for model initialization, training-strategy provenance, and baselines, not as an unverified theorem that forces the conclusion. The one load-bearing circularity risk is the ViDoRe training/evaluation provenance: the same-named corpus appears on both sides, and no disjoint split is documented. Because visual document retrieval is one of the three headlined contributions and the claimed 35% improvement over jina-clip-v1 rests entirely on this benchmark, the provenance gap makes the central claim partially circular as reported. A strict train/test disjointness statement would remove the circularity; absent that, the score is 6 rather than 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper contributes a new model, not a derivation, so the ledger lists the hand-set data processing and training choices that the benchmark claims rest on. The most consequential is the ViDoRe training/evaluation overlap risk.

free parameters (4)
  • SigLIP filtering threshold = ~50% kept (top cosine similarity)
    Hand-chosen cutoff for keeping image-text pairs from DFN/CommonPool; directly shapes the training data used in stages 1 and 2 (Section 3.1).
  • Image resolutions per stage = 224 -> 384 -> 512
    Chosen by the authors; ablation in Section 5.1 shows resolution strongly affects ViDoRe performance, so this choice is load-bearing for the visual document retrieval claim.
  • GPT-4v caption set size = 40,000 images, ~1.2M captions
    Hand-selected number of images and languages for synthetic long captions in Dmm;l (Section 3.1).
  • Caption length threshold for Dmm;l = >256 tokens
    Hand-chosen boundary separating long from short caption datasets (Section 3.1).
assumptions (4)
  • domain assumption The ViDoRe synthetic training data (Section 3.1) does not overlap with the documents or queries used in the ViDoRe benchmark evaluation (Section 4.3).
    Unstated. If false, the reported 52.65 nDCG@5 on ViDoRe is inflated by train/test leakage.
  • domain assumption GPT-4v generated captions are semantically accurate and serve as valid positive text for image-text contrastive learning.
    Used to build Dmm;l; no human verification reported (Section 3.1).
  • domain assumption The multilingual SigLIP ranker's cosine similarities are a reliable proxy for image-text alignment across the filtered languages.
    Used to select the top 50% of DFN/CommonPool pairs; filter quality is not independently evaluated (Section 3.1).
  • domain assumption MTEB and other benchmark implementations, as used by the authors, are correct and match the official versions.
    They rely on MTEB v1.34.7 and CLIP Benchmark; the exclusion of some tasks for bugs/time is reasonable but a caveat (Appendix Table 10 footnote).

how reviews work

0 comments
Cite this review

Pith. "Pith review of jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images." pith.science (2026). https://pith.science/paper/ROMSPYG2

@misc{pith2026241208802,
  author       = {Pith},
  title        = {Pith review of: jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROMSPYG2}},
  note         = {Machine review of arXiv:2412.08802}
}
read the original abstract

Contrastive Language-Image Pretraining (CLIP) has been widely used for crossmodal information retrieval and multimodal understanding tasks. However, CLIP models are mainly optimized for crossmodal vision-language tasks and underperform in single-mode text tasks. Moreover, these models are often trained on English datasets and therefore lack multilingual understanding. Additionally, from a visual understanding perspective, previous CLIP-based models exhibit insufficient understanding of visually rich documents. In this work, we propose jina-clip-v2, a contrastive vision-language model trained on text pairs, triplets and image-text pairs via a multi-task and multi-stage contrastive learning paradigm in order to support both text-only and crossmodal tasks. We employ a multilingual text encoder and expand the training dataset to include multilingual texts from 29 non-English languages, including Hindi, Chinese, German, French, and others, as well as images of visually rich documents. We evaluate the model's performance and show that jina-clip-v2 achieves notable improvements over state-of-the-art CLIP-based models in zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval tasks in both English and multilingual settings. jina-clip-v2 also provides for flexibility in embedding dimensionality, enabling users to select the granularity of the representations. jina-clip-v2 is publicly available at https://huggingface.co/jinaai/jina-clip-v2.

Figures

Figures reproduced from arXiv: 2412.08802 by the authors.

Figure 1
Figure 1. jina-clip-v2 combines a text encoder (Jina XLM-RoBERTa, 561M parameters) and a vision encoder (EVA02-L14, 304M parameters) for a total of 865M parameters. (Muennighoff et al., 2023). Despite its strengths, jina-clip-v1 (Koukounas et al., 2024) has several limitations. First, it is an English-only model, which makes it unsuitable for multilingual document retrieval. Second, jina-clip-v1 struggles with visually-rich i… view at source ↗
Figure 2
Figure 2. Performance on the ViDoRe benchmark (Faysse et al., 2024) against input resolution [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 2
Figure 2. Unsurprisingly, increasing image resolution has a positive impact on linking queries to visually rich documents. The most significant improvement occurs when resolution increases from (224, 224) to (384, 384), with the average nDCG@5 score across 10 benchmarks rising from 0.256 to 0.454. A further increase in resolution to (512, 512) also brings a noticeable gain in performance. Considering that the model is trained… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Contrastive learning between two embedding groups (Unified Batch technique). The first [PITH_FULL_IMAGE:figures/full_fig_p015_3.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    MMArt offers 74,234 paintings with four complementary perspectives and a unified caption, and shows the perspectives are task-asymmetric.

  2. Illuminating Visual Identity in Universal Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.

  3. KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A 2B single-vector retriever, KoVRE, trained with hard-negative mining and reranker distillation, achieves top aggregate Korean visual document retrieval scores.

  4. MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A human-annotated benchmark with 4,800 QA pairs evaluates how well AI models can retrieve and generate interleaved text-and-image answers.

Reference graph

Works this paper leans on

50 extracted references · 18 canonical work pages · cited by 4 Pith papers

  1. [3]

    Sharegpt4v: Improving large multi-modal models with better captions, 2023a

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023a. URLhttps: //arxiv.org/abs/2311.12793. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Intern...

    arXiv

  2. [6]

    Marta R Costa-juss`a, James Cross, Onur C ¸ elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al

    URL https://arxiv.org/ abs/1911.02116. Marta R Costa-juss`a, James Cross, Onur C ¸ elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672,

    arXiv 1911

  3. [7]

    Sedigheh Eslami and Gerard de Melo

    URLhttps://arxiv.org/abs/ 2205.14135. Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Investigating approaches for improving cross-modal alignment in clip. arXiv preprint arXiv:2406.17639,

    arXiv

  4. [8]

    Data filtering networks, 2023a

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks, 2023a. URL https://arxiv.org/abs/2309.17425. Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A-02: A Visual Representation for Neon Genesis.arXiv preprint arXiv:2303.11331, 2023b. URL https: //arxiv...

    arXiv

  5. [9]

    URL https://arxiv.org/abs/2407.01449. 10 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim En- tezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yo...

    arXiv 2025

  6. [13]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  7. [14]

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi

    URL https://arxiv.org/abs/2405.20204. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning,

    arXiv

  8. [15]

    URLhttps://arxiv.org/abs/2205.13147. Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable trans- former modelling library. https://github.com/fac...

    arXiv

Show all 50 references
  1. [16]

    doi: 10.18653/v1/ 2024.acl-long.775

    Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.775. URL https://aclanthology.org/2024.acl-long.775. Shengzhi Li and Nima Tajbakhsh. Scigraphqa: A large-scale synthetic multi-turn question- answering dataset for scientific graphs,

    2024 doi

  2. [17]

    11 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping

    URL https://arxiv.org/abs/2203.02053. 11 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms,

    2025 arXiv

  3. [18]

    Ilya Loshchilov and Frank Hutter

    URL https: //arxiv.org/abs/2411.02571. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization,

    arXiv

  4. [19]

    Minesh Mathew, Viraj Bagal, Rub `en P´erez Tito, Dimosthenis Karatzas, Ernest Valveny, and C

    URL https: //arxiv.org/abs/1711.05101. Minesh Mathew, Viraj Bagal, Rub `en P´erez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021a. URL https://arxiv.org/abs/2104.12756. Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A datas...

    2007 arXiv

  5. [20]

    URL https://arxiv.org/abs/2210.07316. OpenAI. Gpt-4v(ision) system card,

    arXiv

  6. [21]

    Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang

    URL https://api.semanticscholar.org/ CorpusID:263218031. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In K...

    2021

  7. [22]

    doi: 10.18653/v1/2021.naacl-main.466

    Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.466. URL https://aclanthology.org/2021.naacl-main.466/. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Cl...

    2021 doi

  8. [23]

    Janarthanan Rajendran, Mitesh M

    URL https://arxiv.org/abs/2103.00020. Janarthanan Rajendran, Mitesh M. Khapra, Sarath Chandar, and Balaraman Ravindran. Bridge correlational neural networks for multilingual multimodal representation learning,

    arXiv

  9. [25]

    Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.arXiv preprint arXiv:2103.01913,

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.arXiv preprint arXiv:2103.01913,

    arXiv

  10. [26]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu

    URL https://arxiv.org/ abs/2409.10173. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding,

    arXiv

  11. [27]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao

    URLhttps://arxiv.org/abs/ 2104.09864. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale,

    arXiv

  12. [28]

    Ashish Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut

    URL https://arxiv.org/abs/2303.15389. Ashish Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. In EMNLP,

    arXiv

  13. [29]

    Van den Oord, Yazhe Li, and Oriol Vinyals

    A. Van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748 ,

    arXiv

  14. [31]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei

    doi: 10.1109/CVPR46437.2021.00252. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multi- lingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672,

    2021

  15. [33]

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang

    URL https://arxiv.org/abs/2303.15343. Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip, 2024a. URL https://arxiv.org/abs/2403.15378. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang...

    arXiv

  16. [34]

    URL http://dx.doi.org/10.1145/3503161.3548422

    doi: 10.1145/3503161.3548422. URL http://dx.doi.org/10.1145/3503161.3548422. 13 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) A A PPENDIX Table 6: Training settings on each stage Parameter Stage 1 Stage 2 Stage 3 Image encoder weights init ...

    2025

  17. [35]

    81.46 77.12 81.74 79.02 79.20 77.76 14 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) Figure 3: Contrastive learning between two embedding groups (Unified Batch technique). The first group concatenates original images, question texts, and on...

    2025

  18. [36]

    Language - Model jina-clip-v2 jina-clip-v2 stage 1 jina-clip-v2 stage 2 jina-clip-v1 nllb-siglip large nllb-siglip base Zero-shot Image Retrieval - Recall@5 [%] average 81.43 73.51 84.13 16.93 82.07 79.29 ar 73.56 66.22 76.89 0.19 78.92 76.94 bn 63.78 50.19 68.11 0.11 75.19 74...

  19. [37]

    Language - Model jina-clip-v2 jina-clip-v2 stage 1 jina-clip-v2 stage 2 jina-clip-v1 nllb-siglip large nllb-siglip base Zero-shot Image Retrieval - Recall@5 [%] average 84.87 80.66 86.11 31.01 87.60 86.23 de 85.70 80.40 86.80 48.80 88.30 87.00 en 89.40 84.00 89.40 89.00 89.40 ...

    2025

  20. [38]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/3) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] AILACasedocs test 32.79 6.62 8.63 34.73 29.21 31.88 23.10 AILAStatutes test 14.45 15.04 13.58 33.00 22.58 26.45 8.86 ARCChallenge test 10.52 5.85 6.78 10.12 11.18 10.92 10.96 AlphaNLI test...

  21. [39]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (2/3) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 LEMBPasskeyRetrieval test 2048 34.00 4.00 4.00 72.00 12.00 14.00 12.00 LEMBPasskeyRetrieval test 4096 72.00 0.00 0.00 42.00 2.00 20.00 0.00 LEMBPasskeyRetrieval test 8192 82.00 0.00 2.00 26.00 10.00 4.00 8.00 L...

  22. [40]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (3/3) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Touche2020 Retrieval.v3 test 57.00 27.64 33.36 55.40 47.28 52.60 56.19 WikipediaRetrieval Multilingual test 90.91 73.24 76.80 91.78 89.89 90.40 91.85 WinoGrande test 49.60 45.80 44.84 19.58 22.21 28.12 43.78 XM...

    2025

  23. [41]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/1) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] BelebeleRetrieval test 15.96 63.34 70.18 91.32 92.23 92.47 91.57 BelebeleRetrieval test 17.04 62.98 68.77 92.24 91.50 91.16 90.81 CmedqaRetrieval dev 1.81 5.09 7.42 35.89 32.48 31.98 31.73...

  24. [42]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/1) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] BelebeleRetrieval test 54.13 39.84 59.47 92.99 93.20 93.21 92.08 GerDaLIR test 2.14 0.14 0.41 16.18 9.06 10.57 2.22 GerDaLIRSmall test 6.18 0.47 1.34 36.66 21.23 25.33 6.76 GermanDPR test ...

  25. [43]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/1) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] BelebeleRetrieval test 62.63 52.86 69.01 93.52 91.81 91.42 92.36 MIRACLRetrieval HardNegatives dev 27.05 16.71 22.04 51.08 43.17 45.33 50.94 MLQARetrieval test 42.28 29.25 40.56 68.62 62.8...

  26. [44]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/1) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] BelebeleRetrieval test 9.30 63.01 71.61 93.11 90.74 90.96 92.41 MIRACLRetrieval HardNegatives dev 1.15 18.44 21.74 65.29 42.54 49.71 59.74 MultiLongDoc Retrieval test 2.85 15.13 20.25 49.5...

    2025

  27. [45]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/1) Task Split jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] ArguAna-PL test 7.46 10.03 22.07 38.52 43.92 46.80 40.45 BelebeleRetrieval test 30.29 35.21 53.19 92.45 89.16 89.63 90.97 DBPedia-PL test 13.01 9.41 13.34 34.88 24.07 24.83 30.45 DBPedia-P...

  28. [46]

    These tasks were excluded either due to bugs in the evaluation code or excessive computation times

    (1/2) Task Split Languages jc-v1 nllb-b nllb-l je-v3 jc-v2-s1 jc-v2-s2 jc-v2 Retrieval nDCG@10 [%] BelebeleRetrieval test de,en 66.96 52.96 69.05 92.38 92.11 92.36 92.40 BelebeleRetrieval test en,de 46.64 35.59 62.72 91.80 90.18 91.19 89.84 BelebeleRetrieval test fr,en 79.80 6...

  29. [47]

    81.46 81.10 81.10 80.70 78.66 73.00 27 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) Table 24: MRL (Kusupati et al.,

    2025

  30. [48]

    Language - Dimension 1024 768 512 256 128 64 Zero-shot Image Retrieval - Recall@5 [%] A verage 81.43 82.35 82.31 81.75 78.17 72.52 ar 73.56 73.39 73.17 72.61 68.42 62.28 bn 63.78 63.67 63.64 62.39 57.58 49.58 da 85.39 85.31 84.67 84.53 81.69 75.69 de 91.25 91.28 91.47 91.47 88...

    2025

  31. [49]

    ablation study on XTD10 (Aggarwal & Kale, 2020; Rajen- dran et al.,

    2020

  32. [50]

    Language - Dimension 1024 768 512 256 128 64 Zero-shot Image Retrieval - Recall@5 [%] A verage 84.87 84.85 84.60 84.32 81.80 77.85 de 85.70 85.40 84.90 84.70 81.10 79.30 en 89.40 89.60 88.70 88.90 86.50 83.00 es 85.90 85.90 86.00 85.70 84.00 80.80 fr 85.10 84.70 84.80 85.30 83...

    2025

  33. [2014]

    URL https://aclanthology.org/Q14-1006

    doi: 10.1162/tacl a 00166. URL https://aclanthology.org/Q14-1006. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on...

    doi

  34. [2015]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm ´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov

    URL http://arxiv.org/abs/1504.00325. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm ´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale,

    arXiv

  35. [2016]

    Simon Schrodi, David T Hoffmann, Max Argus, V olker Fischer, and Thomas Brox

    URL https://arxiv.org/abs/1510.03519. Simon Schrodi, David T Hoffmann, Max Argus, V olker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- language representation learning. arXiv preprint arX...

    arXiv

  36. [2018]

    URL http://arxiv.org/abs/1807. 03748. 12 To appear at the ICLR 2025 Workshop on Open Science for Foundation Models (SCI-FM) Alexander Visheratin. Nllb-clip–train performant multilingual image retrieval model on a budget. arXiv preprint arXiv:2309.01859, 2023a. Alexander Visher...

    2025 arXiv

  37. [2019]

    doi: 10.18653/v1/K19-1049

    Association for Computational Linguistics. doi: 10.18653/v1/K19-1049. URL https://aclanthology. org/K19-1049/. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models,

    doi

  38. [2020]

    Rethinking benchmarks for cross-modal image-text retrieval

    Weijing Chen, Linli Yao, and Qin Jin. Rethinking benchmarks for cross-modal image-text retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pp. 1241–1251, New York, NY , USA, 2023b. Associatio...

  39. [2021]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    URL https: //arxiv.org/abs/2106.09685. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conferenc...

    arXiv

  40. [2022]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge dis- tillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge dis- tillation. arXiv preprint arXiv:2402.03216,

    arXiv

  41. [2023]

    Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano

    URL https://arxiv.org/ abs/2304.14108. Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. Learning dense representations for entity retrieval. In Mohit Bansal and Aline Villavicencio (eds.), Proceedings of the...

    arXiv

  42. [2024]

    Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren

    URL https://arxiv.org/ abs/2305.13035. Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingual and multi- lingual clip. In Proceedings of the thirteenth language resources and evaluation conference, pp. 6848–6854,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.