Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read 560K carefully synthesized samples outperform 26M prior ones and set state-of-the-art scores on a 36-task multimodal benchmark.

desk verdict mmE5 is a well-executed synthetic-data recipe that lifts multimodal embedding performance, but the headline data-efficiency claim is clouded by an untested train/test image-overlap route and a backbone mismatch. read the letter →

arxiv 2502.08468 v1 pith:757UIOYW submitted 2025-02-12 cs.CV cs.AIcs.CL

Haonan Chen , Liang Wang , Nan Yang , Yutao Zhu , Ziliang Zhao , Furu Wei , Zhicheng Dou This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords multimodalembeddingsmultilingualsyntheticdatasynthesisqualitycontrastivelearningvision-languagemodelsretrievalzero-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal embedding models, which map images and text into one vector space, are held back by the scarcity of labeled data. This paper argues that synthetic data can close that gap if it meets three criteria: broad scope across tasks, modality combinations, and languages; strong cross-modal alignment between texts and images; and high fidelity through real images and self-checked text. To test this, the authors build a one-pass, self-evaluating synthesis pipeline and train mmE5 on 560K synthetic samples. On the 36-task MMEB benchmark, mmE5 reaches 58.6 zero-shot and 69.8 supervised, beating a prior model trained on 26M synthetic samples, and it sets the best average Recall@10 on the seven-language XTD benchmark. If the result holds, data quality rather than data quantity is the lever that makes universal multimodal multilingual embeddings practical.

What carries the argument

The load-bearing object is the one-pass, self-evaluating data-synthesis pipeline. It combines four steps, data configuration, multi-aspect visual interpretation (general description, object-level details, contextual features, and task-specific brainstorming), text synthesis, and self-evaluation with revision, inside a single call to a multimodal large language model. The single-pass design preserves image context and cross-modal alignment, while the self-evaluation step, which scores relevance, plausibility, clarity, and diversity before revising, keeps fidelity high. The other key element is the data quadruple (task instruction, query, positive document, hard negative document, where a hard negative is a document that looks relevant but is wrong), which feeds a contrastive InfoNCE loss during finetuning.

What would settle it

Re-run the exact 560K pipeline with an open-source multimodal large language model whose pretraining data are known to exclude the MMEB and XTD image sets, and compare zero-shot scores; if the open-source teacher reproduces the 58.6 average, the criteria carry the gain, while a large drop would implicate benchmark overlap in the proprietary teacher.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quality of synthetic multimodal training data can be defined and engineered. Its framework first configures each sample by task (classification, VQA, or retrieval), modality combination (seven types such as image-to-text, text-to-image, and image-text-to-image-text), language (93 languages), and real query images drawn from an open image corpus. A multimodal large language model then produces a multi-aspect visual interpretation, writes the instruction, query, positive document, and hard-negative document, evaluates its own output on relevance, plausibility, clarity, and diversity, and rewrites the sample, all in one pass so the model continuously sees the image. The resulting 560K samples are used with a contrastive InfoNCE loss to finetune mmE5. Trained this way, mmE5 scores 58.6 zero-shot and 69.8 supervised on MMEB, compared with 44.0 for a model trained on 26M synthetic samples, and it averages 95.3 Recall@10 across seven languages on XTD.

Load-bearing premise

The result assumes that the proprietary model generating the synthetic data describes the sampled images honestly and has not already absorbed the evaluation benchmarks through its own training data, so the reported gains reflect the design criteria rather than teacher memorization.

Editorial extensions

If this is right

  • Synthetic data quality, not sheer volume, is the main driver: 560K samples produced by the three-criteria pipeline outperform 26M samples from prior pipelines on average MMEB score.
  • Coverage across three tasks and seven modality combinations lets a single model generalize to classification, VQA, retrieval, and grounding; dropping any task type lowers MMEB performance in ablations.
  • Multilingual synthetic data transfers to English benchmarks about as well as English-only data while adding strong gains on non-English retrieval, and the 93-language set is enough to set a state of the art on XTD.
  • The synthesis recipe transfers to different base multimodal large language models, improving each over its own baseline, so the method is not tied to one backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The pipeline does not test whether the proprietary teacher model has already seen the evaluation benchmarks, so part of the reported gap could come from teacher memorization rather than the three design criteria; a clean check would replace the teacher with an open model whose pretraining data exclude those benchmarks.
  • Inference: The three criteria could double as a filtering or selection signal for existing synthetic corpora, not just a generation prompt, meaning candidate samples could be scored for scope, alignment, and fidelity and pruned without regenerating them.
  • Inference: The same criteria appear modality-agnostic, so the single-pass self-evaluating design may transfer to audio and video embeddings, where labeled data are even scarcer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes mmE5, a multimodal multilingual embedding model trained on 560K synthetic samples produced by a single-pass GPT-4o pipeline. The pipeline samples real images from LAION-400M, performs multi-aspect visual interpretation, generates task/query/positive/hard-negative tuples across three tasks, seven modality combinations, and 93 languages, and then self-evaluates and refines each sample. The authors report state-of-the-art results on the 36-task MMEB benchmark (zero-shot 58.6 versus MMRet's 44.0 with 26M synthetic samples; supervised 69.8) and best average Recall@10 on the seven-language XTD benchmark (95.3). They attribute the gains to three data-quality criteria: broad scope, robust cross-modal alignment, and high fidelity.

Significance. If the results hold up under contamination controls, this is a valuable contribution: it suggests that a relatively small, carefully synthesized multimodal dataset can outperform much larger synthetic corpora, and the authors support this with extensive ablations (Table 5), a scaling analysis (Figure 4), same-backbone comparisons (Table 4), and detailed prompt documentation in the appendix. The public release of code, datasets, and models is a further strength. The main risks are empirical rather than logical: possible train/test image overlap between LAION-400M samples and MMEB/XTD test images, and a headline comparison that confounds data scale with backbone strength. Both are testable and should be addressed before the central claims can be accepted.

major comments (3)
  1. The training images are sampled from LAION-400M without any exclusion of images that appear, or are near-duplicates of, images in the MMEB or XTD test sets. MMEB includes MSCOCO, ImageNet, VisualNews, and other web-sourced images that are very likely present in LAION-400M. If such images enter the 560K training set, the contrastive objective can memorize image-to-text associations that transfer directly to test queries, and the GPT-4o generation step can produce text that is close to original captions or annotations. The paper reports no overlap statistics and no training run on an exclusion-based subset, even though the Limitations section acknowledges only the proprietary-teacher issue. This route is plausible and directly affects the headline 58.6 zero-shot MMEB score and the 45x data-efficiency claim. Please add a quantitative overlap analysis (e.g., nearest-neighbor or perceptual-hash similarity between synthetic training images and MMEB/XTD test images), report results after training on an overlap-excluded subset, and, if feasible, ablate the teacher model to separate image-overlap effects from synthesis-criteria effects.
  2. The zero-shot comparison in Table 2 compares mmE5 built on Llama-3.2-11B-Vision against MMRet built on Phi-3.5-V, so the stated 45x-data advantage is not a controlled comparison of data quality. Table 4 demonstrates that mmE5 beats VLM2Vec and MMRet baselines when compared on the same base MLLM, but it does not include MMRet on Llama-3.2-11B-Vision or an equivalent 26M-sample baseline on the same backbone. The central data-efficiency claim therefore needs a same-backbone, same-data-budget experiment, or an explicit reformulation that the improvement is measured across backbones.
  3. The hyperparameter analysis is described as follows: 'we tune these hyperparameters on evaluation datasets comprising 1K samples from each training set. However, for consistency with previous experiments, we present results on the MMEB test sets.' This makes it unclear whether any MMEB test labels were used for model selection. If the LoRA rank, batch size, and temperature in Figure 5 were chosen using test-set performance, the reported zero-shot averages are optimistically biased. Please clarify the split used for tuning and, if test labels were used, provide a proper held-out validation split and re-report the affected numbers.
minor comments (5)
  1. The header 'w/o. Visual Interpertation' contains a misspelling of 'Interpretation'.
  2. The notation for modality combinations is inconsistent: the text and Figure 1 use arrows (IT→I, IT→IT, T→IT), while Section 4.5.1 and Table 5 use 'IT2I', 'IT2IT', and 'T2IT'. Please unify the notation.
  3. The row label 'w/ english synthetic data' should be capitalized as 'w/ English synthetic data'.
  4. The x-axis is labeled 'Synthetic Data Size (log scale)', but the plotted ticks appear linear (50K, 100K, 200K, ...). Please clarify whether the axis is logarithmic and adjust the tick labels accordingly.
  5. The claim of using '45 times less training data' should explicitly say '45x less synthetic data' and should note that the comparison in Table 2 is across different backbone models; the current wording implies a controlled data-efficiency comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark evaluation is external and the synthesis claims are supported by ablations, not by construction.

full rationale

I find no circular derivation chain. The paper's central empirical claims—SOTA on MMEB and XTD—are established by evaluation on external benchmarks, not by construction from the synthesis pipeline. The three data-quality criteria are asserted and then tested through ablations (Table 5) and scaling experiments (Figure 4); no reported score is algebraically forced by the synthesis equations. The self-citations to the authors' prior E5-style synthesis work (Wang et al. 2024a; Chen et al. 2024) supply prompt/data-format conventions, but the load-bearing evidence for the method is the external benchmark comparison and component ablations, so these citations are not circular. The acknowledged reliance on GPT-4o and the absence of an overlap analysis between LAION-400M training images and MMEB/XTD test images are legitimate contamination and reproducibility risks, but they are not cases where a prediction reduces to its inputs by definition. The note that hyperparameters are tuned on 1K-sample evaluation splits and then reported on MMEB test sets is a methodology detail worth scrutiny, but it is standard validation selection rather than a fitted quantity renamed as a prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The 'three criteria' are design heuristics, not entities. The free parameters listed are hand-chosen design decisions that affect the reported benchmark numbers; the axioms are standard domain assumptions for empirical synthetic-data training.

free parameters (4)
  • Synthetic data task ratio = 1:1:2 (classification:VQA:retrieval)
    Chosen by hand in Section 4.1; retrieval is weighted more because it has more modality combinations. This ratio shapes the zero-shot MMEB average and could be tuned to favor retrieval.
  • Language sampling weights = English 36.4%, 14 languages about 3.1% each (Fig. 3)
    Section 3.2.1 states high-source languages get higher weights, with no exact formula; the resulting distribution affects multilingual and English performance.
  • Hard negative mining rank = 70th position in VLM2Vec-LoRA ranking
    Appendix B.2: hard negatives for labeled finetuning are selected at rank 70; the choice affects contrastive difficulty and downstream scores.
  • Training hyperparameters (batch size, temperature, LoRA rank) = LoRA rank 8, lr 2e-5, one epoch; batch size and temperature selected via Fig. 5 ranges
    Section 4.1 and 4.6: these settings are tuned on 1K-sample subsets of each MMEB training set and affect the reported MMEB and XTD scores.
assumptions (4)
  • domain assumption InfoNCE contrastive learning on synthetic (t, q, d+, d-) quadruples transfers to the MMEB and XTD benchmarks.
    Section 3.3; the entire evaluation rests on this transfer, standard in the field but unproven beyond the reported results.
  • domain assumption GPT-4o-generated texts are semantically aligned with the LAION images and do not systematically leak benchmark answers.
    Sections 3.2.2 and 4.1; the quality-criteria framework assumes the teacher MLLM produces faithful, non-contaminated labels.
  • domain assumption Average Precision@1 over 36 MMEB tasks and Recall@10 over 7 XTD languages is a valid measure of embedding quality.
    Section 4.1; the SOTA claim is relative to these aggregated metrics, which can hide per-task failures (e.g., retrieval underperformance).
  • domain assumption LAION-400M images are a suitable real-world image source for synthesis.
    Section 3.2.1; the high-fidelity criterion assumes the corpus is clean enough and representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data." pith.science (2026). https://pith.science/paper/757UIOYW

@misc{pith2026250208468,
  author       = {Pith},
  title        = {Pith review of: mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/757UIOYW}},
  note         = {Machine review of arXiv:2502.08468}
}
read the original abstract

Multimodal embedding models have gained significant attention for their ability to map data from different modalities, such as text and images, into a unified representation space. However, the limited labeled multimodal data often hinders embedding performance. Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck. In this work, we identify three criteria for high-quality synthetic multimodal data. First, broad scope ensures that the generated data covers diverse tasks and modalities, making it applicable to various downstream scenarios. Second, robust cross-modal alignment makes different modalities semantically consistent. Third, high fidelity ensures that the synthetic data maintains realistic details to enhance its reliability. Guided by these principles, we synthesize datasets that: (1) cover a wide range of tasks, modality combinations, and languages, (2) are generated via a deep thinking process within a single pass of a multimodal large language model, and (3) incorporate real-world images with accurate and relevant texts, ensuring fidelity through self-evaluation and refinement. Leveraging these high-quality synthetic and labeled datasets, we train a multimodal multilingual E5 model mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art performance on the MMEB Benchmark and superior multilingual performance on the XTD benchmark. Our codes, datasets and models are released in https://github.com/haon-chen/mmE5.

Figures

Figures reproduced from arXiv: 2502.08468 by the authors.

Figure 1
Figure 1. An illustration of our data synthesis frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of our method. We take the generation of an IT [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of languages in the synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: The zero-shot performances of mmE5 with different training settings on MMEB (280K synthetic data for efficient test). 1K samples from each training set. However, for consistency with previous experiments, we present results on the MMEB test sets. LoRA Rank denotes the …
Figure 6
Figure 6. Figure 6: An example of the synthetic Retrieval IT2IT data (part 1). This part includes the input images, the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: An example of the synthetic Retrieval IT2IT data (part 2). This part includes the evaluation, possible [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: An example of the synthetic Classification IT2T data. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: An example of the synthetic VQA IT2T data. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Douyin Multimodal Embedding Model Technical Report

    cs.IR 2026-08 conditional novelty 6.0 of 10

    Latent typed reasoning plus cross-conditional reconstruction during training improves multimodal retrieval accuracy while keeping inference a standard dense bi-encoder, yielding 74.8 (2B) and 78.4 (9B) on MMEB-v2.

  2. Illuminating Visual Identity in Universal Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.

  3. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 unverdicted novelty 6.0 of 10

    FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.

  4. MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A two-stage training recipe that converts causal VLMs into bidirectional multimodal embedding models, achieving SOTA on MMEB.

  5. Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.

  6. Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

    cs.IR 2026-03 conditional novelty 5.0 of 10

    CoCoA forces an MLLM to reconstruct masked text through a single EOS token, improving multimodal embedding quality on MMEB-V1 and matching MoCa at 3B with far less pretraining data.

  7. Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Visual token compression (4x fewer tokens) plus a three-stage generative/contrastive/judge-curated training pipeline yields state-of-the-art MLLM-based retrieval accuracy at lower inference cost.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages · cited by 7 Pith papers

  1. [1]

    Relevance: Are the generated input texts and labels (if not empty) tightly connected to their corresponding image and task objectives? Does the task instruction effectively link the query image with the positive label?

    work page

  2. [2]

    Plausibility: Are misleading labels sufficiently relevant to the image or labels while remaining definitively incorrect? Could they mislead the model?

    work page

  3. [3]

    Clarity: Is the generated task clear and unambiguous, providing sufficient instruction to connect the query image with the label, without being overly specific or abstract?

    work page

  4. [4]

    description

    Diversity: Does the generated data introduce variation in task instructions, texts (if not empty), and labels to avoid repetitive patterns in the dataset? - Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths. - Suggest specific revisions to address any identified weaknesses, ensur...

    work page

  5. [6]

    Relevance: Are the generated question and answers tightly linked to the image content and consistent with the task requirements?

    work page

  6. [7]

    hard_negative_answer

    Plausibility: Does the "hard_negative_answer" closely resemble the "positive_answer" while remaining definitively incorrect? Could it mislead the model?

    work page

  7. [8]

    question

    Diversity: Does the generated data introduce variation in questions, and answers to avoid repetitive patterns in the dataset? - Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths. - Suggest specific revisions to address any identified weaknesses, ensuring the revised data better a...

    work page

  8. [9]

    Relevance: Are the generated query and documents (if not empty) tightly connected to their corresponding image and task objectives? Does the task instruction effectively link the query image with the positive text?

    work page

Show all 16 references
  1. [10]

    Plausibility: Are hard negatives sufficiently similar to the query or positive examples while remaining definitively incorrect? Could they mislead the model?

  2. [11]

    Clarity: Is the generated task clear and unambiguous, providing sufficient instruction to connect the query image with the positive document, without being overly specific or abstract?

  3. [12]

    description

    Diversity: Does the generated data introduce variation in task instructions, queries, and documents to avoid repetitive patterns in the dataset? - Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths. - ...

  4. [13]

    Relevance: Are the generated query and documents (if present) tightly linked to their corresponding images? Does the task instruction effectively connect the query image to the positive image?

  5. [14]

    Plausibility: Are the negative examples, including hard negatives, realistic and similar enough to the positive image to challenge the model, while still being definitively incorrect?

  6. [15]

    Clarity: Is the generated task clear and unambiguous, providing sufficient instruction to connect the query image with the positive image, without being overly specific or abstract?

  7. [16]

    For each criterion, explain specific flaws or strengths

    Diversity: Does the generated data introduce variation in task instructions, queries, and documents to avoid repetitive patterns in the dataset? - Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths. - ...

  8. [2024]

    input_text

    Uniir: Training and benchmarking univer- sal multimodal information retrievers. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Pro- ceedings, Part LXXXVII, volume 15145 of Lecture Notes in Computer Science, pages 387–404....

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.