Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single 7B model can outperform larger specialized retrieval models across over 40 multimodal tasks through careful data curation and a modality-aware contrastive loss.

desk verdict Solid empirical recipe for universal multimodal embedders, but the load-bearing data-curation claims rest on small-scale ablations that are never validated at the final 7B/6.4M scale. read the letter →

arxiv 2505.19650 v2 pith:NQS6LYOA submitted 2025-05-26 cs.CV cs.IRcs.MM

Fanheng Kong , Jingyuan Zhang , Yahui Liu , Hongzhi Zhang , Shi Feng , Xiaocui Yang , Daling Wang , Yu Tian
show 3 more authors
Victoria W. Fuzheng Zhang Guorui Zhou
This is my paper
classification cs.CVcs.IRcs.MM
keywords multimodalretrievaluniversalembeddingsdatacurationmodality-awaremaskedcontrastivelearningMAMCLvision-languagemodelsvideo-textinstruction-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UNITE is a training framework for a single multimodal embedding model that handles text, image, video, and fused queries. The paper's central claim is that with carefully curated modality data and a modality-aware contrastive loss, a 7B model can outperform larger specialized retrieval models across more than 40 tasks spanning coarse-grained, fine-grained, and instruction-based retrieval. The authors report that text-video pairs are the most effective training data for general cross-modal retrieval, while text-text and text-image pairs are essential for instruction following. They also find that fine-grained video-caption pairs should be added directly to retrieval adaptation rather than used in a separate alignment stage. The proposed loss, MAMCL, restricts contrastive comparisons to candidates with the same target modality, and this mask is what lets the model balance text, image, and video in one embedding space.

What carries the argument

The central mechanism is Modal-Aware Masked Contrastive Learning (MAMCL), a masked variant of the InfoNCE loss. For each query, the similarity matrix is multiplied by a modality mask $\mathbf{M}$ whose entry is 1 only when the candidate's target modality combination matches that of the positive candidate; mismatched entries are set to $-\infty$, so the query is contrasted only against candidates of the same target modality. This removes the competitive interference the paper observes when text-only, image-only, video-only, and fused embeddings are contrasted jointly. The second pillar is the data curation recipe derived from 600K-sample composition experiments: roughly 22% text-text, 39% image-text, and 36% video-text pairs in retrieval adaptation, followed by instruction tuning on MMEB plus WebVid-CoVR.

What would settle it

Train the final 7B model on a 6.4M-sample dataset in which the text-video fraction is replaced by text-image data while keeping total size and hyperparameters fixed; if image-text retrieval does not drop relative to UNITE, the paper's claim that video-text pairs are superior training data for image-text retrieval is contradicted.

Watch

Extended reading notes

Core claim

The paper claims that the modality composition of training data is a first-order determinant of performance in multimodal retrieval, and that a single language-multimodal model can reach state-of-the-art results across task families if that composition is chosen correctly. The empirical discoveries are: text-video pairs alone beat text-image pairs even on image-text retrieval; text-text and text-image pairs are needed for instruction-based retrieval; and direct inclusion of fine-grained video captions in retrieval adaptation is both more effective and cheaper than a separate fine-grained alignment stage. Built on Qwen2-VL and trained on 6.4M retrieval-adaptation samples plus 1.3M instruction-tuning samples, UNITE reports leading Recall@1 numbers on CaReBench, ShareGPT4V, Urban1K, DOCCI, WebVid-CoVR, and the 36-task MMEB benchmark, surpassing models with 11B and 26B parameters.

Load-bearing premise

The load-bearing premise is that the data-composition conclusions from 600K-sample experiments on a 2B model still hold for the final 6.4M-sample, 7B training run, a transfer the paper assumes without scaling validation.

Editorial extensions

If this is right

  • A single 7B embedder can replace separate text-image, text-video, and composed-retrieval models, so downstream systems can serve many retrieval tasks with one index and one model.
  • Multimodal data recipes should assign a substantial share to video-text pairs even when the target benchmark is image-text, because the paper finds TV-only training outperforms TI-only training on that task.
  • Fine-grained video caption data should be folded into retrieval adaptation rather than staged as a separate alignment pass; the paper reports this is both more accurate and about four times cheaper per 500K instances.
  • MAMCL's modality mask is a general plug-in: any new modality can be added by defining its mask from input prior knowledge, extending the framework beyond text, image, and video.
  • Model scaling helps most on temporal video retrieval, where the 7B model shows its largest relative gains over the 2B model, suggesting temporal reasoning is a scale-hungry capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The data-composition conclusions come from 600K-sample runs on a 2B backbone; the paper's final recipe assumes these conclusions transfer to 6.4M samples and a 7B model, a scaling step the paper does not directly validate.
  • The TV-over-TI result for image-text retrieval may reflect differences in caption quality or dataset difficulty between InternVid and CapsFusion rather than modality per se; an ablation that matches caption length and cleanliness would isolate the cause.
  • MAMCL could be extended to mask by fine-grained subtype, such as spatial versus temporal video aspects, or by task family, which the paper does not explore and which might sharpen instruction-based retrieval further.
  • If the recipe transfers, the field's emphasis on ever-larger retrieval models would shift toward data composition and loss design, making the paper's contribution a blueprint for adding audio or other modalities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents UNITE, a unified multimodal embedder built on Qwen2-VL (2B/7B) with a two-stage training scheme: retrieval adaptation on 6.4M curated text-text, image-text, and video-text pairs, followed by instruction tuning on 1.3M samples from MMEB and WebVid-CoVR. The authors propose Modal-Aware Masked Contrastive Learning (MAMCL), which restricts contrastive comparisons to candidates sharing the same target modality, to reduce inter-modal interference. They report extensive evaluations on fine-grained, coarse-grained, and instruction-based retrieval benchmarks, claiming state-of-the-art results on 40+ tasks. A systematic study in Section 5 examines how data composition (TT, TI, TV, and mixtures) affects downstream performance, and the final recipe is justified by those findings.

Significance. If substantiated, the paper's main contribution is an empirical recipe for curating multimodal training data and a modality-aware contrastive loss that generalizes to arbitrary modalities. The paper is valuable for its breadth of evaluation (MMEB, WebVid-CoVR, CaReBench, Flickr30K, etc.), transparent hyperparameters (LoRA rank, temperature, batch sizes), and the inclusion of OOD splits within MMEB, which provides some independent grounding. The data-composition analysis is a useful starting point, though the conclusions are drawn from small-scale ablations. The MAMCL idea is simple and plausible, but the measured gains are modest. Overall the work is potentially a useful engineering contribution, but the central causal claims are stronger than the current evidence.

major comments (3)
  1. [§5, Appendix B.3, Tables 9/18/19, Figure 5] The data-composition conclusions central to the 'modality curation' claim are drawn entirely from 600K-sample ablations on Qwen2-VL-2B. The final 6.4M-sample, 7B runs use a different composition (Figure 5: TT 21.6%, TI 39.2%, TV 36.1%) and never test whether the configuration ordering from Table 9 persists at that scale. For example, Table 9 shows TV-only outperforming TI-only on Flickr30K T→I (60.2 vs 55.4) and the full TT+TI+TV mix underperforming TV-only on CaRe V→T (61.2 vs 68.7), yet the final recipe includes all three modalities. The differences between some configurations are under one point (e.g., MMEB overall 61.1 vs 62.1; CoVR R@1 65.6 vs 64.8) with no variance or multiple seeds. Consequently, the attribution of UNITE's SOTA results to the curated composition is not directly supported; the results could be driven by scale, backbone, or other training details. This should be addressed by either validating the composition at the target scale (e.g., a 7B/6.4M ablation of at least the key configurations) or substantially softening the causal claim.
  2. [§4.1, Tables 4 and 20, Abstract] The claim of 'state-of-the-art in 40+ different tasks' is broader than the evidence. On the MMEB meta-task level, UNITE 7B's Grounding score (84.8) is below CAFe 7B (91.2), mmE5 11B (89.7), and IDMR 26B (88.6). On individual datasets in Table 20, UNITE 7B is not the best on ImageNet-1K (80.2 vs IDMR 80.6), VOC2007 (84.9 vs IDMR 92.7), N24News (80.3 vs mmE5 81.7), or RefCOCO (89.2 vs mmE5 92.7), among others. The overall MMEB average (70.3) is within 0.5 points of mmE5 and CAFe (69.8). Thus the superiority claim should be qualified to specific benchmarks/tasks, and the 40+ count should be verified against the per-dataset tables.
  3. [§4.2, Table 7] The MAMCL ablation shows modest gains: for the 7B model, adding MAMCL increases MMEB overall by +0.3 and the MMEB/CoVR average by +0.7; for 2B, +0.5 and +1.1. No standard deviations or multiple-seed results are reported, and OOD changes are within ±0.5. The qualitative claim that MAMCL 'mitigates competitive relationships' is therefore not strongly quantified. If MAMCL is a core contribution, the authors should either provide statistical significance/variance estimates or present a more measured interpretation.
minor comments (5)
  1. [§3.1] There is a typo: 'repsentation space' should be 'representation space'.
  2. [§5] There is a typo: 'instrction-based' should be 'instruction-based'.
  3. [Author list] The author list contains 'Victoria W.' without a full surname; this appears to be a placeholder and should be corrected.
  4. [Limitations] The Limitations paragraph discusses audio as a future modality but does not acknowledge that the data-composition conclusions in Section 5 are based on 600K/2B experiments; given that the core claim rests on these conclusions, this limitation should be stated explicitly.
  5. [Table 4] Table 4 mixes zero-shot and fine-tuned results in one table; the 'Finetuning Setting' rows are not directly comparable to the 'Zero-shot Setting' rows, and the caption should clarify which baselines are fine-tuned on which data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UNITE's contributions are empirical and self-contained; MAMCL and the data-curation recipe are evaluated on held-out benchmarks and are not defined in terms of the scores they claim to predict.

full rationale

The paper's central claims are empirical rather than definitional. MAMCL is introduced as a loss modification (Eq. 3–5) whose mask depends only on input modality type, not on any benchmark score; its effect is then tested through ablations on held-out evaluation sets. The data-composition conclusions in Section 5 are obtained from 600K-sample, Qwen2-VL-2B experiments and are used to construct the final training mixture. This is a design choice informed by ablations, not a fitted parameter renamed as a prediction, and the paper does not claim to predict the final scores from those ablations. The MMEB benchmark is used both for instruction tuning and evaluation, but this is a standard supervised setting, and the paper separately reports results on external benchmarks (CaReBench, WebVid-CoVR, Flickr30K, MSR-VTT, MSVD, DiDeMo, DOCCI, ShareGPT4V, Urban1K) that were not used for training. Self-citations to the authors' prior work ([37], [107], [110]) appear only in the related-work and reference lists and are not load-bearing for the main argument. The lack of direct validation that the 600K/2B data-composition conclusions transfer to the 6.4M/7B setting is a legitimate scaling and generalization concern, but it is not circularity: the final model's performance is not derived by construction from the ablation numbers. No circular step can be quoted or reduced to the paper's own equations, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard contrastive learning, an assumed effective pooling prompt, and the paper's own data-mixing choices. There are no invented physical or conceptual entities. The main unverified premise is that small-scale data-composition findings transfer to the full-scale model.

free parameters (3)
  • Temperature tau = 0.03
    Used in both contrastive stages; a standard hyperparameter chosen by hand, not fit to the target benchmarks.
  • LoRA rank and alpha = rank 8, alpha 16/64
    Hyperparameters for parameter-efficient fine-tuning, chosen without reported tuning.
  • Training data proportions = TT ~21.6%, TI ~39.2%, TV ~39.2% (approx.)
    Chosen based on the paper's own data-composition analysis, which was run at a smaller scale (600K samples on a 2B model) and assumed to transfer to the final 6.4M/7B setup.
assumptions (3)
  • standard math InfoNCE contrastive loss with cosine similarity provides a valid learning signal for retrieval.
    Used throughout as the base objective, following prior work.
  • domain assumption The EOL prompt 'Summarize above ... in one word:' yields an embedding that captures the semantics of any input modality combination.
    Adopted from PromptEoL and E5-V; the paper does not test alternative pooling or prompt strategies.
  • domain assumption Masking contrastive pairs to the same target modality reduces inter-modal interference and improves retrieval.
    Supported by ablations in Table 7, but the effect is small and the mechanism is not directly measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval." pith.science (2026). https://pith.science/paper/NQS6LYOA

@misc{pith2026250519650,
  author       = {Pith},
  title        = {Pith review of: Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQS6LYOA}},
  note         = {Machine review of arXiv:2505.19650}
}
read the original abstract

Multimodal information retrieval (MIR) faces inherent challenges due to the heterogeneity of data sources and the complexity of cross-modal alignment. While previous studies have identified modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce UNITE, a universal framework that tackles these challenges through two critical yet underexplored aspects: data curation and modality-aware training configurations. Our work provides the first comprehensive analysis of how modality-specific data properties influence downstream task performance across diverse scenarios. Moreover, we propose Modal-Aware Masked Contrastive Learning (MAMCL) to mitigate the competitive relationships among the instances of different modalities. Our framework achieves state-of-the-art results on multiple multimodal retrieval benchmarks, outperforming existing methods by notable margins. Through extensive experiments, we demonstrate that strategic modality curation and tailored training protocols are pivotal for robust cross-modal representation learning. This work not only advances MIR performance but also provides a foundational blueprint for future research in multimodal systems. Our project is available at https://friedrichor.github.io/projects/UNITE.

Figures

Figures reproduced from arXiv: 2505.19650 by the authors.

Figure 1
Figure 1. Performance comparison on instruction-based retrieval benchmarks (left: MMEB [ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of UNITE: (a) Model architecture utilizing LMM as the backbone, supporting multimodal inputs (text, images, videos, and their combinations). (b) Similarity matrix after applying MAMCL, which enables focused contrastive learning by restricting comparisons to samples sharing the same target modality, thus reducing inter-modal interference. where <vision> and <text> are placeholders for visual content (i.e., i… view at source ↗
Figure 3
Figure 3. We develop a universal multimodal embedder [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance comparison on fine-grained video-text benchmark (CaReBench [ [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Retrieval Adaptation 6.4M. Left: Data Distribution within Each Category. The outer circle shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Right: The detailed quantities of datasets. Instruction Tuning 1.3 M Ima…
Figure 6
Figure 6. Figure 6: Instruction Tuning 1.3M. Left: Data Distribution within Each Category. The outer circle shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Right: The detailed quantities of datasets. A.2 Evaluation Datasets We pro…

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

    cs.CV 2026-08 conditional novelty 7.0 of 10

    UniME-R1 uses a failure-aware adviser to diagnose embedding mistakes from initial retrieval results and then either reranks candidates or re-retrieves with a feedback-based query rewrite.

  2. SOLAR: Self-supervised Joint Learning for Symmetric Multimodal Retrieval

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SOLAR learns symmetric MM2MM embeddings self-supervised by disentangling image-text intersection masks then using them to auto-generate contrastive samples, outperforming 10 SOTA models including large VLMs on a new 2...

  3. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

    cs.CV 2026-04 unverdicted novelty 6.5 of 10

    Using a structured rewrite instead of CoT as the generative interface improves MLLM-based multimodal embedding performance while cutting thinking tokens by about half.

  4. Illuminating Visual Identity in Universal Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.

  5. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 unverdicted novelty 6.0 of 10

    FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.

  6. TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TUNA introduces a 1,000-video benchmark with dense temporal captions and 1,432 multiple-choice questions, and finds that current video LMMs are weakest at camera motion, action sequences, and multi-subject scenes.

Reference graph

Works this paper leans on

120 extracted references · 30 canonical work pages · cited by 6 Pith papers

  1. [1]

    Multimodal automated fact-checking: A survey

    Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. Multimodal automated fact-checking: A survey. InFindings of the Conference on Empirical Methods in Natural Language Processing, 2023. 2

    2023

  2. [2]

    Localizing moments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017. 7, 19, 21, 25

    2017

  3. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2015. 2

    2015

  4. [4]

    Finelip: Extending clip’s reach via fine- grained alignment with longer text inputs.arXiv preprint arXiv:2504.01916, 2025

    Mothilal Asokan, Kebin Wu, and Fatima Albreiki. Finelip: Extending clip’s reach via fine- grained alignment with longer text inputs.arXiv preprint arXiv:2504.01916, 2025. 6

    arXiv 2025

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 2

    arXiv 2025

  6. [6]

    Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016. 5, 19, 22

    arXiv 2016

  7. [7]

    Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  8. [8]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark.arXiv preprint arXiv:2410.03051, 2024

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq-Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark.arXiv preprint arXiv:2410.03051, 2024. 9

    arXiv

Show all 120 references
  1. [9]

    Webqa: Multihop and multimodal qa

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 19

    2022

  2. [10]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL),

  3. [11]

    mme5: Improving multimodal multilingual embeddings via high-quality synthetic data

    Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. arXiv preprint arXiv:2502.08468, 2025. 6, 7, 24, 26

    2025 arXiv

  4. [12]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision (ECCV), 2024. 18, 20

    2024

  5. [13]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. 3

    2023

  6. [14]

    Murag: Multimodal retrieval-augmented generator for open question answering over images and text.arXiv preprint arXiv:2210.02928, 2022

    Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal retrieval-augmented generator for open question answering over images and text.arXiv preprint arXiv:2210.02928, 2022. 3

    2022 arXiv

  7. [15]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821,

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821,

    arXiv

  8. [16]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF Conference on Computer Vision ...

    2023

  9. [17]

    Visual dialog

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 19

    2017

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 19

    2009

  11. [19]

    Mmdocir: Benchmarking multi-modal retrieval for long documents.arXiv preprint arXiv:2501.08828,

    Kuicai Dong, Yujing Chang, Xin Deik Goh, Dexun Li, Ruiming Tang, and Yong Liu. Mmdocir: Benchmarking multi-modal retrieval for long documents.arXiv preprint arXiv:2501.08828,

  12. [20]

    The pascal visual object classes challenge: A retrospective.International Journal of Computer Vision, 111:98–136, 2015

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective.International Journal of Computer Vision, 111:98–136, 2015. 19

    2015

  13. [21]

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344, 2023

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344, 2023. 19

    2023 arXiv

  14. [22]

    Simcse: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. 2, 5, 19, 22

    2021

  15. [23]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3

    2023

  16. [24]

    Breaking the modality barrier: Universal embedding learning with multimodal llms.arXiv preprint arXiv:2504.17432, 2025

    Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal embedding learning with multimodal llms.arXiv preprint arXiv:2504.17432, 2025. 2, 6, 7, 24, 26

    2025

  17. [25]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. 21

    2022

  18. [26]

    Egocvr: An egocentric benchmark for fine-grained composed video retrieval

    Thomas Hummel, Shyamgopal Karthik, Mariana-Iuliana Georgescu, and Zeynep Akata. Egocvr: An egocentric benchmark for fine-grained composed video retrieval. InEuropean Conference on Computer Vision (ECCV), 2024. 3, 6

    2024

  19. [27]

    Mate: Meet at the embedding-connecting images with long texts

    Young Kyun Jang, Junmo Kang, Yong Jae Lee, and Donghyun Kim. Mate: Meet at the embedding-connecting images with long texts. InFindings of the Conference on Empirical Methods in Natural Language Processing, 2024. 6

    2024

  20. [28]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational Conference on Machine Learning (ICML), 2021. 1, 3

    2021

  21. [29]

    Tencent text- video retrieval: hierarchical cross-modal interactions with multi-level representations.IEEE Access, 2022

    Jie Jiang, Shaobo Min, Weijie Kong, Hongfa Wang, Zhifeng Li, and Wei Liu. Tencent text- video retrieval: hierarchical cross-modal interactions with multi-level representations.IEEE Access, 2022. 3

    2022

  22. [30]

    Scaling sentence embeddings with large language models

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. InFindings of the Conference on Empirical Methods in Natural Language Processing, 2024. 3 11

    2024

  23. [31]

    E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580, 2024

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580, 2024. 2, 3, 6, 24

    2024 arXiv

  24. [32]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks.arXiv preprint arXiv:2410.05160, 2024

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks.arXiv preprint arXiv:2410.05160, 2024. 1, 2, 3, 4, 6, 7, 18, 20, 22, 24, 25, 26

    2024 arXiv

  25. [33]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017. 5

    2017

  26. [34]

    Visual question answering: Datasets, algorithms, and future challenges.Computer Vision and Image Understanding, 163:3–20, 2017

    Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future challenges.Computer Vision and Image Understanding, 163:3–20, 2017. 2

    2017

  27. [35]

    Deep visual-semantic alignments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 20

    2015

  28. [36]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. InAdvances in Neural Information Processing Systems (NeurIPS), 2020. 19

    2020

  29. [37]

    Tuna: Comprehensive fine-grained temporal understanding evaluation on dense dynamic videos.arXiv preprint arXiv:2505.20124, 2025

    Fanheng Kong, Jingyuan Zhang, Hongzhi Zhang, Shi Feng, Daling Wang, Yu Tian, Linhao Yu, Xingguang Ji, Victoria W, and Fuzheng Zhang. Tuna: Comprehensive fine-grained temporal understanding evaluation on dense dynamic videos.arXiv preprint arXiv:2505.20124, 2025. 3

    2025 arXiv

  30. [38]

    Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research.Transactions of the Association for Computat...

    2019

  31. [39]

    Llave: Large language and vision embedding models with hardness-weighted contrastive learning.arXiv preprint arXiv:2503.04812, 2025

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness-weighted contrastive learning.arXiv preprint arXiv:2503.04812, 2025. 3

    2025

  32. [40]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024. 3

    2024 arXiv

  33. [41]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning (ICML), 2023. 2, 6, 24

    2023

  34. [42]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational Conference on Machine Learning (ICML), 2022. 3

    2022

  35. [43]

    Video- llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video- llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 3

    2023 arXiv

  36. [44]

    Mm-embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024. 3, 5, 24

    2024 arXiv

  37. [45]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014. 5, 19, 22, 25

    2014

  38. [46]

    Idmr: Towards instance-driven precise visual correspondence in multimodal retrieval.arXiv preprint arXiv:2504.00954, 2025

    Bangwei Liu, Yicheng Bao, Shaohui Lin, Xuhong Wang, Xin Tan, Yingchun Wang, Yuan Xie, and Chaochao Lu. Idmr: Towards instance-driven precise visual correspondence in multimodal retrieval.arXiv preprint arXiv:2504.00954, 2025. 3, 6, 7, 24, 26 12

    2025 arXiv

  39. [47]

    Visual news: Benchmark and challenges in news image captioning.arXiv preprint arXiv:2010.03743, 2020

    Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning.arXiv preprint arXiv:2010.03743, 2020. 19

    2010 arXiv

  40. [48]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  41. [49]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. 2, 3

    2024

  42. [50]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

    2024

  43. [51]

    Llava-plus: Learning to use tools for creating multimodal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. InEuropean Conference on Computer Vision (ECCV), 2024. 3

    2024

  44. [52]

    Lamra: Large multimodal model as your advanced retrieval assistant.arXiv preprint arXiv:2412.01720, 2024

    Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant.arXiv preprint arXiv:2412.01720, 2024. 2, 3, 7, 25

    2024 arXiv

  45. [53]

    Image retrieval on real-life images with pre-trained vision-and-language models

    Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2, 3, 19

    2021

  46. [54]

    Generative multi-modal knowledge retrieval with large language models

    Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. Generative multi-modal knowledge retrieval with large language models. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024. 3

    2024

  47. [55]

    End-to-end knowl- edge retrieval with multi-modal queries

    Man Luo, Zhiyuan Fang, Tejas Gokhale, Yezhou Yang, and Chitta Baral. End-to-end knowl- edge retrieval with multi-modal queries. InProceedings of the Annual Meeting of the Associa- tion for Computational Linguistics (ACL), 2023. 3

    2023

  48. [56]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024. 3

    2024

  49. [57]

    Unifying multimodal retrieval via document screenshot embedding

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. 3

    2024

  50. [58]

    Videogpt+: Integrating image and video encoders for enhanced video understanding.arXiv preprint arXiv:2406.09418,

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding.arXiv preprint arXiv:2406.09418,

    arXiv

  51. [59]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2, 19

    2019

  52. [60]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Annual Meeting of the Association for Computational Linguistics, 2022. 19

    2022

  53. [61]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022. 19

    2022

  54. [62]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021. 3, 19

    2021

  55. [63]

    Mm1: methods, analysis and insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mm1: methods, analysis and insights from multimodal llm pre-training. InEuropean Conference on Computer Vision (ECCV), 2024. 2 13

    2024

  56. [64]

    Docci: Descriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. InEuropean Conference on Computer Vision (ECCV),

  57. [65]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 3, 4

    2018 arXiv

  58. [66]

    Kosmos- g: Generating images in context with multimodal large language models.arXiv preprint arXiv:2310.02992, 2023

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos- g: Generating images in context with multimodal large language models.arXiv preprint arXiv:2310.02992, 2023. 3

    2023 arXiv

  59. [67]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE/CVF International Conference on Computer Vis...

    2015

  60. [68]

    Filtering, distillation, and hard negatives for vision-language pre-training

    Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. InProceedings of the IEEE/CVF Conference on Computer V...

    2023

  61. [69]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICM...

    2021

  62. [70]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016. 5, 19

    2016

  63. [71]

    Contrastive learning with hard negative samples.arXiv preprint arXiv:2010.04592, 2020

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples.arXiv preprint arXiv:2010.04592, 2020. 8

    2010 arXiv

  64. [72]

    Pic2word: Mapping pictures to words for zero-shot composed image retrieval

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3

    2023

  65. [73]

    Laion- 5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. InAdvances in Neural Info...

    2022

  66. [74]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision (ECCV), 2022. 19

    2022

  67. [75]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023. 6

    2023 arXiv

  68. [76]

    Composed video retrieval via enriched context and discriminative embeddings

    Omkar Thawakar, Muzammal Naseer, Rao Muhammad Anwer, Salman Khan, Michael Fels- berg, Mubarak Shah, and Fahad Shahbaz Khan. Composed video retrieval via enriched context and discriminative embeddings. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  69. [77]

    Fever: a large-scale dataset for fact extraction and verification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 809–81...

    2018

  70. [78]

    Covr-2: Automatic data construction for composed video retrieval.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr-2: Automatic data construction for composed video retrieval.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 5, 6 14

    2024

  71. [79]

    Covr: Learning composed video retrieval from web video captions

    Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr: Learning composed video retrieval from web video captions. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024. 1, 2, 3, 6, 18, 19, 20, 22

    2024

  72. [80]

    Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024. 5, 9

    2024 arXiv

  73. [81]

    A comprehensive survey on cross-modal retrieval.arXiv preprint arXiv:1607.06215, 2016

    Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang. A comprehensive survey on cross-modal retrieval.arXiv preprint arXiv:1607.06215, 2016. 3

    2016 arXiv

  74. [82]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2, 3, 5, 21, 22

    2024 arXiv

  75. [83]

    Fvqa: Fact-based visual question answering.IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2413–2427, 2017

    Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering.IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2413–2427, 2017. 2

    2017

  76. [84]

    Cross- modal retrieval: a systematic review of methods and future directions.Proceedings of the IEEE, 2025

    Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. Cross- modal retrieval: a systematic review of methods and future directions.Proceedings of the IEEE, 2025. 3

    2025

  77. [85]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. InInternational Conference on Learning Representations (ICLR), 2023. 5...

    2023

  78. [86]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. InEuropean Conference on Computer Vision (ECCV), 2024. 2, 5

    2024

  79. [87]

    Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022. 7, 25

    2022 arXiv

  80. [88]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025. 3

    2025 arXiv

  81. [89]

    Omnibind: Large-scale omni multimodal representation via binding spaces.arXiv preprint arXiv:2407.11895, 2024

    Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Heng- shuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces.arXiv preprint arXiv:2407.11895, 2024. 3

    2024 arXiv

  82. [90]

    N24news: A new dataset for multimodal news classification

    Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24news: A new dataset for multimodal news classification. InProceedings of the Language Resources and Evaluation Conference (LREC), 2022. 19

    2022

  83. [91]

    Uniir: Training and benchmarking universal multimodal information retrievers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision (ECCV), 2024. 3, 24

    2024

  84. [92]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2010. 19

    2010

  85. [93]

    Approximate nearest neighbor negative contrastive learning for dense text retrieval.arXiv preprint arXiv:2007.00808, 2020

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval.arXiv preprint arXiv:2007.00808, 2020. 8

    2007 arXiv

  86. [94]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 7, 19, 20, 22, 25 15

    2016

  87. [95]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024. 3

    2024 arXiv

  88. [96]

    Fine-grained video-text retrieval: A new benchmark and method.arXiv preprint arXiv:2501.00513, 2024

    Yifan Xu, Xinhao Li, Yichun Yang, Rui Huang, and Limin Wang. Fine-grained video-text retrieval: A new benchmark and method.arXiv preprint arXiv:2501.00513, 2024. 5, 7, 9, 18, 19, 22, 23, 25

    2024 arXiv

  89. [97]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018. 5, 19

    2018 arXiv

  90. [98]

    End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models

    Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retri...

    2023

  91. [99]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 5

    2024 arXiv

  92. [100]

    Cafe: Unifying representation and generation with contrastive-autoregressive finetuning.arXiv preprint arXiv:2503.19900, 2025

    Hao Yu, Zhuokai Zhao, Shen Yan, Lukasz Korycki, Jianyu Wang, Baosheng He, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, and Hanchao Yu. Cafe: Unifying representation and generation with contrastive-autoregressive finetuning.arXiv preprint arXiv:2503.19900, 2025. 6, 7, 24

    2025 arXiv

  93. [101]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022. 3

    2022 arXiv

  94. [102]

    Capsfusion: Rethinking image-text data at scale

    Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5, 19, 22

    2024

  95. [103]

    Tarsier2: Advanc- ing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advanc- ing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025. 5, 9, 22

    2025 arXiv

  96. [104]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 24

    2023

  97. [105]

    Long-clip: Unlocking the long-text capability of clip

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. InEuropean Conference on Computer Vision (ECCV), 2024. 5, 6, 18, 19, 20, 25

    2024

  98. [106]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning.arXiv preprint arXiv:2409.20566, 2024. 2

    2024 arXiv

  99. [107]

    Data metabolism: An efficient data design schema for vision language model.arXiv preprint arXiv:2504.12316, 2025

    Jingyuan Zhang, Hongzhi Zhang, Zhou Haonan, Chenxi Sun, Jiakang Wang, Fanheng Kong, Yahui Liu, Qi Wang, Fuzheng Zhang, et al. Data metabolism: An efficient data design schema for vision language model.arXiv preprint arXiv:2504.12316, 2025. 2

    2025 arXiv

  100. [108]

    Magiclens: Self-supervised image retrieval with open-ended instructions.arXiv preprint arXiv:2403.19651, 2024

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions.arXiv preprint arXiv:2403.19651, 2024. 2, 3, 6, 7, 24

    2024 arXiv

  101. [109]

    Gme: Improving universal multimodal retrieval by multimodal llms.arXiv preprint arXiv:2412.16855, 2024

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms.arXiv preprint arXiv:2412.16855, 2024. 2, 3, 24

    2024 arXiv

  102. [110]

    Stickerconv: Generating multimodal empathetic responses from scratch

    Yiqun Zhang, Fanheng Kong, Peidong Wang, Shuang Sun, SWangLing SWangLing, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song. Stickerconv: Generating multimodal empathetic responses from scratch. InProceedings of the Annual Meeting of the Association for Computational Lingui...

    2024

  103. [111]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. 5

    2024

  104. [112]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 9

    2024 arXiv

  105. [113]

    Retrieving multimodal informa- tion for augmented generation: A survey

    Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal informa- tion for augmented generation: A survey. InFindings of the Conference on Empirical Methods in Natural Lang...

    2023

  106. [114]

    Megapairs: Massive data synthesis for universal multimodal retrieval.arXiv preprint arXiv:2412.14475, 2024

    Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval.arXiv preprint arXiv:2412.14475, 2024. 6, 7, 24, 26

    2024 arXiv

  107. [115]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023. 5, 6, 7, 25

    arXiv

  108. [116]

    Deep learning for video-text retrieval: a review.International Journal of Multimedia Information Retrieval, 12(1):3, 2023

    Cunjuan Zhu, Qi Jia, Wei Chen, Yanming Guo, and Yu Liu. Deep learning for video-text retrieval: a review.International Journal of Multimedia Information Retrieval, 12(1):3, 2023. 3

    2023

  109. [117]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 2

    2023 arXiv

  110. [118]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 2

    2025 arXiv

  111. [119]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 19

    2016

  112. [120]

    Apollo: An exploration of video understanding in large multimodal models.arXiv preprint arXiv:2412.10360, 2024

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models.arXiv preprint arXiv:2412.10360, 2024. 2, 3 17 Image ...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.