{"id":"6cb510ca-bf2e-43ab-897f-ed7d92446256","arxiv_id":"2412.08802","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"jina-clip-v2, an 865M-parameter multilingual dual-encoder, outperforms prior CLIP-style models on text-only and crossmodal retrieval, and on visually rich document retrieval, while supporting flexible embedding dimensions.","lead":"jina-clip-v2 is a new open-weight model that aligns text and images in 30 languages, improving multilingual retrieval and visual document search over earlier CLIP-style models. It also lets users truncate its 1,024-dimension embeddings down to 256 dimensions with under 1% performance loss.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ViDoRe benchmark results may be inflated by training on ViDoRe synthetic data; the paper never demonstrates a disjoint split, so the headline 52.65 nDCG@5 and 35% gain over jina-clip-v1 are not yet trustworthy.","rationale":"The reader's weakest assumption identifies exactly the load-bearing fault in the paper. The central claim is that a single model can serve text-only, crossmodal, and visually rich document retrieval across languages. The first two parts are supported by MTEB and crossmodal tables that, while not perfect, are plausible and do not involve an identically named training source. The visual document retrieval part rests entirely on the ViDoRe benchmark, and the paper openly trains on 'ViDoRe synthetic training data'. Without an explicit statement that the benchmark's test queries and documents were excluded, the reported 35% improvement over jina-clip-v1 and the absolute 52.65 nDCG@5 are unverifiable. This is not a claim about author intent; it is a structural provenance gap that a simple overlap analysis would close. I keep the CONDITIONAL verdict because the other contributions may stand, but the paper should not be fully accepted until the ViDoRe disjointness is demonstrated.","tokens_in":36898,"tokens_out":6288,"duration_ms":63492,"concrete_test":"Download the released ViDoRe synthetic training data and the ViDoRe benchmark test collections (10 datasets). Compute exact and near-duplicate overlap between (a) each training image and each benchmark image using perceptual hashing, MinHash on OCR text, or CLIP cosine similarity > 0.9, and (b) each synthetic query and each benchmark query using normalized exact matching and MinHash. Report the fraction of benchmark query-document pairs having a near-duplicate in training. If the fraction is nonzero, recompute Table 4 averages on the non-overlapping subset; if the average drops by more than about 1 nDCG@5 point, the headline ViDoRe claim is overstated. If the fraction is zero, the concern is resolved, but the paper should still state the disjointness explicitly. A stronger follow-up is to retrain a checkpoint without the ViDoRe synthetic data and measure the ViDoRe delta.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1 states that Dmm;s and Dmm;l include 'ViDoRe synthetic training data (Faysse et al., 2024)', and Section 4.3 reports results on the ViDoRe benchmark (Table 4, average nDCG@5 52.65%, a 35% improvement over jina-clip-v1). The paper never states that the benchmark's test queries and documents were excluded from, or are disjoint from, this training data. Because the training set and the benchmark share the same name and were produced by the same authors, the natural null hypothesis is that they derive from the same source PDFs, possibly with overlapping or near-duplicate pages and queries. If that is true, the reported ViDoRe scores reflect train/test leakage rather than a genuine capability gain. This is load-bearing because visual document retrieval is one of the three headlined contributions and the ViDoRe result is the only direct evidence for it. The same provenance gap applies to the other visual-document datasets listed in Section 3.1 (DocVQA, TatDQA, InfographicsVQA, SciGraphQA, ArXivQA/ArXivCAP, WIT): the paper does not document that benchmark test instances were held out. The absence of any deduplication or split-provenance statement makes the 35% improvement unverifiable as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces jina-clip-v2, an 865M-parameter dual-encoder model that combines a multilingual Jina-XLM-RoBERTa text encoder and an EVA02-L/14 vision encoder. Training uses a multi-task, multi-stage contrastive objective with InfoNCE losses on text pairs, text triplets with hard negatives, and image-text pairs, including multilingual data and visually rich document images. The model is evaluated on English and multilingual crossmodal retrieval, MTEB retrieval and STS tasks, the ViDoRe visual document retrieval benchmark, and an ablation of Matryoshka Representation Learning. The central claims are that jina-clip-v2 outperforms prior CLIP-based models on multilingual crossmodal and text-only tasks, achieves a 35% improvement over jina-clip-v1 on ViDoRe, and retains performance when embeddings are truncated to 256 dimensions.","tokens_in":37173,"tokens_out":3439,"duration_ms":34167,"significance":"If the reported results are trustworthy, jina-clip-v2 is a practically valuable open-weight model: it provides a single encoder for text-only, crossmodal, and visually rich document retrieval across many languages, with transparent training-stage details, public model weights, and a Matryoshka-style dimensionality trade-off that loses under 1% performance at 256 dimensions. The explicit multi-stage training recipe and the analysis of resolution effects are useful additions to the CLIP literature. The main significance risk is that the headline ViDoRe result rests on a train/test provenance gap, and the MTEB text-only comparison has a related provenance question; both need to be resolved before the central claims can be taken at face value.","major_comments":[{"comment":"The ViDoRe benchmark result is load-bearing for the visual document retrieval contribution, but the paper does not establish that the benchmark test set is disjoint from the training data. Section 3.1 states that Dmm;s and Dmm;l include 'ViDoRe synthetic training data (Faysse et al., 2024)', and Section 4.3 reports an average nDCG@5 of 52.65% and a 35% improvement over jina-clip-v1 on the ViDoRe benchmark. Because the ViDoRe synthetic training data is derived from the same benchmark project, and the paper gives no statement of excluded queries or documents or of deduplication, the reported gain could partly reflect train/test leakage. Please provide a precise description of the split between the synthetic training data and the benchmark test collections, report any deduplication performed, and if overlap exists, re-evaluate without the contaminated training data.","section":"§3.1 and §4.3"},{"comment":"The resolution analysis in Section 5.1 is affected by the same provenance issue. The four additional runs are trained 'using the same visually rich training set' and then evaluated on the ViDoRe benchmark; since that training set includes ViDoRe synthetic data, the nDCG@5 curve in Figure 2 may partially measure the model's ability to memorize training-page images rather than the effect of resolution on generalization. Please rerun the resolution check on a training set that provably excludes ViDoRe benchmark documents, or explicitly argue why overlap cannot affect the relative ranking across resolutions.","section":"§5.1"},{"comment":"The text-retrieval evaluation has a parallel provenance gap. Dtxt;p and Dtxt;t are described as 'data in 30 languages ... introduced as training data for jina-embeddings-v3', and Section 4.2 evaluates on MTEB retrieval and STS tasks. Since many MTEB retrieval datasets (e.g., MS MARCO, CQADupstack, FEVER) are standard training sources for text embedding models, the paper should state whether any MTEB test split was included in Dtxt;p or Dtxt;t, and if so, report results on the held-out subsets. Without this, the comparison against jina-embeddings-v3 on retrieval and STS is hard to interpret.","section":"§3.1 and §4.2"}],"minor_comments":[{"comment":"The phrase 'toping jina-clip-v1' should be corrected to 'topping jina-clip-v1'.","section":"§4.3"},{"comment":"The sentence 'compared jina-clip-v1' is missing the word 'to'; it should read 'compared to jina-clip-v1'.","section":"§1, contribution list"},{"comment":"Table 1 states '89 languages supported' for the text encoder while the abstract and Section 3.1 refer to 30 languages in the training data; please clarify whether 89 refers to the underlying XLM-RoBERTa vocabulary support and 30 to the actual training languages, and state this explicitly in the text.","section":"Table 1"},{"comment":"The summation notation 'kP' appears to be a rendering error for the sum over k items; please fix the typesetting so the denominator is unambiguous.","section":"Equation (1)"},{"comment":"The NLLB-CLIP technical report is cited twice as Visheratin (2023a) and Visheratin (2023b); these should be consolidated into a single reference entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The ViDoRe contamination issue is the main risk to the paper's central claim. If the authors can provide a convincing split-provenance or deduplication statement, the revision is likely addressable; if not, the 52.65% nDCG@5 and the 35% improvement claim should be removed or heavily qualified. The MTEB provenance question should also be required as part of the revision, since the paper currently gives no dataset-level overlap analysis for either evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Give you the short version first: jina-clip-v2 is a well-built, openly released 865M dual encoder, and most of its central claims probably hold. The text-only retrieval and crossmodal results across 30 languages are real improvements over jina-clip-v1 and competitive with NLLB-CLIP. The Matryoshka ablation is complete and convincing: dropping to 256 dimensions costs less than 1% on the tasks they report. What I do not trust is the headline ViDoRe number (52.65 nDCG@5, a 35% gain over v1). Section 3.1 lists 'ViDoRe synthetic training data (Faysse et al., 2024)' among the pretraining sources, and Section 4.3 evaluates on the ViDoRe benchmark without ever stating that the benchmark's test queries and documents were held out. The natural null is leakage, and the paper does nothing to rule it out. That is load-bearing: visual document retrieval is one of three headlined improvements, and ViDoRe is the only direct evidence. The same provenance gap affects the other visual-document datasets listed in Section 3.1 (DocVQA, TatDQA, InfographicsVQA, etc.), so I would treat all of Section 4.3 as unverified until the authors either drop those sources or prove disjointness.\n\nCredit where it's due: the training recipe is transparent, the losses are standard InfoNCE with the hard-negative extension clearly specified, and the resolution study in Section 5.1 is a decent piece of analysis. They also honestly state that jina-embeddings-v3 beats them on text retrieval. The report is a competent engineering release, not a research breakthrough, but it doesn't claim to be one.\n\nSecondary soft spots, all minor: no error bars or multiple seeds (some stage-to-stage differences are ~1 point); no comparison to MM-Embed despite citing it; and the stage-3 crossmodal drop is attributed to a trade-off hypothesis without a test. These don't change my overall read.\n\nBottom line: if you want a single open model for multilingual text and image embeddings, this is a solid option, and the paper is worth a serious referee. But the ViDoRe leakage question must be answered before the visual-document-retrieval claim can be believed. I'd send it to review with that as the required revision.","headline":"Useful multilingual dual encoder with a solid text/crossmodal core, but the headline ViDoRe gain is unverified due to likely train/test leakage.","tokens_in":37758,"tokens_out":2535,"would_cite":true,"duration_ms":24805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single 865M-parameter embedding model can retrieve text, images, and visually rich documents across 30 languages, and its vectors can be cut to 256 dimensions with under 1% quality loss.","keywords":["multilingual embeddings","contrastive learning","crossmodal retrieval","visual document retrieval","Matryoshka representation learning","zero-shot retrieval","text embeddings","vision-language model"],"falsifier":"Retrain jina-clip-v2 without the ViDoRe-derived synthetic data and re-evaluate on the ViDoRe test tasks; if the reported 35% improvement over jina-clip-v1 collapses when that split is removed, the gain is partly a benchmark-leakage artifact rather than a genuine visual-document capability.","tokens_in":36699,"feed_emoji":"🌐","tokens_out":7140,"duration_ms":66755,"temperature":0.7,"pith_summary":"The paper sets out to show that one open-weight embedding model can cover text-only retrieval, semantic similarity, image-to-text and text-to-image retrieval, and visually rich document retrieval across many languages. Its central move is to extend the jina-clip-v1 recipe, which jointly optimizes text-text and text-image contrastive objectives, with a multilingual text encoder, escalating image resolution, and Matryoshka Representation Learning. The authors report that jina-clip-v2 outperforms previous CLIP-style models on multilingual crossmodal retrieval, text retrieval, and STS, and lands close to the dedicated text embedder jina-embeddings-v3 on text-only benchmarks. If true, the practical consequence is that one model can replace several separate embedding systems, and its vectors can be truncated to a quarter of their size with negligible quality loss.","feed_headline":"One 865M model now handles text, image, and document retrieval","feed_subtitle":"It matches dedicated text embedders on retrieval and STS and keeps 99% of full-size quality when cut to 256 of 1,024 dimensions.","key_machinery":"The load-bearing mechanism is a multi-task, multi-stage contrastive training loop. At every stage the model minimizes an InfoNCE loss, a contrastive objective that pulls paired query-target embeddings together and pushes in-batch negatives apart, and stage 3 adds a hard-negative extension that pits each query against seven explicit negatives. The text encoder is initialized from the multilingual Jina-XLM-RoBERTa checkpoint and the image encoder from EVA02-L/14, and the stages progressively raise text context length from 77 to 512 tokens and image resolution from 224 to 512 pixels. In addition, every loss is recomputed on truncated embeddings of 64, 128, 256, 512, and 768 dimensions, which is what trains the model so that smaller output vectors remain useful.","core_discovery":"The paper's central claim is that a dual-encoder model, jina-clip-v2, trained with a multi-task, multi-stage contrastive objective, can act as both a multilingual text embedder and a multilingual vision-language embedder without sacrificing either role. The authors report that it outperforms prior CLIP-style models on zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval in English and across 29 non-English languages, and that on the MTEB retrieval and STS suites it lands close to the dedicated multilingual text embedder jina-embeddings-v3. They further claim that the model understands visually rich documents such as tables, graphs, infographics, and PDF-style pages far better than jina-clip-v1, reaching 52.65 average nDCG@5 on the ViDoRe benchmark, and that its 1,024-dimension embeddings can be truncated to 256 dimensions with under 1% loss thanks to Matryoshka Representation Learning.","pith_inferences":["If the ViDoRe synthetic training examples overlap the ViDoRe benchmark's test documents, part of the reported 35% visual-document improvement may reflect benchmark leakage rather than a new capability; checking the split provenance would settle this.","The slight crossmodal drop from stage 2 to stage 3, while text retrieval improves, hints that a single shared embedding space has a ceiling for jointly optimizing both objectives, so task-specific adapters or separate heads may be a natural next step.","The 256-dimension truncation claim is demonstrated only on the benchmarks in this paper; a harder test would be truncating embeddings during out-of-distribution retrieval on new languages and document types."],"forward_implications":["One deployment can serve English and multilingual text retrieval, STS, and both directions of image-text retrieval, replacing separate text and vision embedding models.","Embeddings truncated from 1,024 to 256 dimensions retain over 99% of full-size performance, cutting storage and bandwidth costs substantially.","Visual document retrieval over tables, infographics, scientific graphs, and PDF-style pages becomes practical with a CLIP-style dual encoder, reaching 52.65 average nDCG@5 on ViDoRe.","Multilingual gains over the English-only predecessor are large: up to 67% higher crossmodal scores, up to 60% higher text retrieval scores, and up to 43% higher STS scores on the reported benchmarks."],"supporting_citations":[{"why":"Supplies the multi-task, multi-stage training recipe and the jina-clip-v1 baseline that jina-clip-v2 extends.","marker":"Koukounas et al. (2024)"},{"why":"Provides the Jina-XLM-RoBERTa text encoder initialization and the jina-embeddings-v3 text-only comparison point.","marker":"Sturua et al. (2024)"},{"why":"Defines the ViDoRe benchmark and contributes the visually rich document training data used to close the visual-document gap.","marker":"Faysse et al. (2024)"},{"why":"Introduces Matryoshka Representation Learning, the technique that makes dimensional truncation with minimal loss possible.","marker":"Kusupati et al. (2024)"},{"why":"Provides the MTEB benchmark suite used for the retrieval and semantic textual similarity evaluations.","marker":"Muennighoff et al. (2023)"},{"why":"Provides the NLLB-CLIP multilingual baselines that the paper compares against on crossmodal and text tasks.","marker":"Visheratin (2023b)"},{"why":"Supplies the SigLIP objective and model variants used both as a baseline and as the multilingual filter for image-text training data.","marker":"Zhai et al. (2023)"}],"fun_headline_variants":["One model for multilingual text and image retrieval","jina-clip-v2 unifies text, image, and document retrieval in 29 languages","Matches text-only embedders and handles images: jina-clip-v2","Cut jina-clip-v2 to 256 dims, keep 99% quality","Multilingual CLIP variant outperforms prior models on text and crossmodal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The visual-document retrieval results assume that the ViDoRe synthetic training examples included in the multimodal datasets do not overlap the ViDoRe benchmark's test queries and documents, because the paper does not state that those test items were excluded during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["One model for multilingual text and image retrieval","jina-clip-v2 unifies text, image, and document retrieval in 29 languages","Matches text-only embedders and handles images: jina-clip-v2","Cut jina-clip-v2 to 256 dims, keep 99% quality","Multilingual CLIP variant outperforms prior models on text and crossmodal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001674,"raw_usage":{"total_tokens":6672,"prompt_tokens":1013,"completion_tokens":5659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":5559}},"tokens_in":629,"tokens_out":5659,"duration_ms":40966,"temperature":1.0,"reasoning_tokens":5559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:31:41.051780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain jina-clip-v2 without the ViDoRe-derived synthetic data and re-evaluate on the ViDoRe test tasks; if the reported 35% improvement over jina-clip-v1 collapses when that split is removed, the gain is partly a benchmark-leakage artifact rather than a genuine visual-document capability.","supporting_citations":[],"review_version":1}