Pith. sign in

REVIEW 3 major objections 5 minor 81 references

Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This position paper claims that multimodal correspondences are inherently many-to-many, and that ignoring this 'multiplicity' destabilizes training, biases evaluation, and degrades dataset quality.

desk verdict A clearly argued position paper that names a real phenomenon and traces it through training, evaluation, and data; the transitivity assumption in Property 1 is too strong but the core thesis survives. read the letter →

arxiv 2505.19614 v2 pith:NDJB5T4G submitted 2025-05-26 cs.LG cs.CV

Sanghyuk Chun , Olga Russakovsky This is my paper
classification cs.LGcs.CV
keywords multimodallearningmultiplicitymany-to-manycorrespondencefalsenegativescontrastivecross-modalretrievaldatasetconstructionaleatoricuncertainty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the one-to-one alignment assumption underlying most current multimodal learning is false: real cross-modal relationships are many-to-many, a property the paper calls multiplicity. It claims multiplicity is not annotation noise but an inevitable consequence of intra-modal variability (the same concept appears in many instances), representational asymmetry (images exhaustively capture a scene while captions select a few salient concepts), and task-dependent alignment (what counts as a match depends on the task). Because current methods ignore it, valid matches are treated as negatives, producing false negatives that destabilize contrastive training, make retrieval benchmarks unreliable, and reduce dataset quality. The paper concludes that multimodal learning should be reframed around multiplicity, calling for multiplicity-aware architectures, training objectives, evaluation protocols, and dataset construction.

What carries the argument

The machinery is the formal definition of multiplicity as a relation R ⊆ X × Y with at least one valid counterpart per item, together with three generative properties: intra-modal variability, representational asymmetry, and task-dependent alignment. The argument's load-bearing component is a gradient analysis of contrastive learning under sparse annotation: if a query x has K valid positives in the batch but only one is annotated, the standard loss's gradient deviates from the ideal multi-positive loss's gradient by Σ_j(p_j − p*_j)g(y_j), causing the model to repel valid matches and producing the modality gap; the deviation grows with K. This same false-negative mechanism is shown to make hard negative mining particularly destructive, and to make Recall@K a misleading evaluation metric, motivating the paper's proposed directions of probabilistic embeddings, conditional/specified representations, and multiplicity-aware dataset filtering. The combinatorial growth of pairwise correspondences (quadratic in dataset size) is also part of the machinery, making exhaustive annotation infeasible.

What would settle it

One could settle the central claim by building a small multimodal dataset with dense human annotation of all valid cross-modal pairs, training the same architecture with the standard one-positive contrastive loss and with the ideal many-positive loss, and comparing their learned embeddings and retrieval performance. If the standard model shows no larger modality gap or semantic fragmentation than the ideal model—or if the supposed false negatives have no effect on retrieval—then multiplicity is not the fundamental training bottleneck the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that for any two modalities X and Y, valid cross-modal relations R ⊂ X × Y are typically many-to-many, so |{y: (x,y) ∈ R}| ≥ 1 rather than exactly 1 as standard practice presumes. The authors identify three sources of multiplicity—intra-modal variability, asymmetry between modalities, and task-dependent alignment—and argue that they make one-to-one alignment infeasible even with perfect data. They then show formally that with sparse annotations (one positive per query), the gradient of the standard contrastive loss pushes the embedding of x away from unannotated true positives, with the discrepancy Σ_j(p_j − p*_j)g(y_j) growing as K, the number of valid matches, increases; hard negative mining exacerbates this. On the evaluation side, they argue that Recall@K treats multiple valid matches as errors and can report perfect scores while ranking is poor, whereas ranking-based metrics such as mAP@R with dense human-verified positives align with human preference. On dataset construction, they argue that filtering by CLIP similarity cannot remove multiplicity because underspecified instances (e.g., 'photo') inherently match many items, and that specificity-based filtering reduces it.

Load-bearing premise

The central argument depends on the assumption that semantic similarity transfers across modalities: if two image-caption pairs share overlapping concepts, then the cross-pairings are also valid positive matches; if this transitivity fails for some task, many alleged false negatives are not false at all.

Editorial extensions

If this is right

  • Every current contrastive pre-training run on web-scale data is trained with systematic false negatives, and the number of false negatives grows roughly quadratically with dataset size.
  • Retrieval benchmarks that report Recall@K can overstate performance; reliable evaluation requires multiple human-verified positives per query and rankings, with mAP@R considerably better aligned to human preference.
  • Dataset construction should move from raw CLIP-similarity filtering to specificity-aware filtering, because underspecified pairs are the main amplifiers of multiplicity.
  • A single deterministic embedding per item cannot faithfully represent a polysemous input; probabilistic embeddings or condition-specified representations are needed to handle input and matching ambiguity.
  • Scaling up models and data will not make multiplicity disappear; once multimodal models become strong enough, the one-to-one assumption will become a visible bottleneck, as label noise in unimodal benchmarks did.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same many-to-many structure should appear in audio-visual event alignment, video-language grounding, and instruction following in robotics, where 'what counts as a match' is task-dependent; the paper's framework offers a template for diagnosing false negatives in those settings.
  • A direct test of the gradient analysis: on a small dataset with dense human annotations, training with the ideal multi-positive contrastive loss should reduce the modality gap and improve retrieval robustness relative to standard training; the paper does not run this experiment.
  • The argument implies that current human evaluation of generated multimodal content (captioning, text-to-image) may be the only reliable protocol, since automatic metrics penalize the very diversity that multiplicity makes valid; this could motivate task-conditioned automatic metrics.
  • If false negatives are a main source of the modality gap, then simple data-side interventions—e.g., augmenting each positive pair with semantically similar neighbors—might improve CLIP-style training as much as architectural changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that multimodal data is inherently many-to-many—a property the authors call 'multiplicity'—and that this is inevitable rather than a side-effect of noise or annotation error. It identifies three sources of multiplicity: intra-modal variability, asymmetries in information density between modalities, and task-dependent alignment. The paper then analyzes consequences across the pipeline: training (contrastive loss and hard negative mining suffer from false negatives), evaluation (retrieval metrics and generation metrics are unreliable), and dataset construction (filtering based on alignment scores may amplify multiplicity). It concludes by advocating multiplicity-aware learning frameworks, evaluation protocols, and dataset construction, pointing to probabilistic embeddings as a promising direction.

Significance. The paper is a clearly written position piece that identifies a real and underappreciated issue in multimodal learning. Its main strengths are the taxonomy of multiplicity sources and the formal gradient analysis in Section 3.1, which correctly shows how sparse annotations misdirect the contrastive gradient when valid unannotated positives are treated as negatives. The concrete empirical numbers from prior human-annotation studies (8.5/17.9 additional positives per query in COCO, and roughly 4.4x more positive matches in the ECCV Caption benchmark) give the argument grounding. However, the paper's central 'inevitability' claim rests on a transitivity assumption (Property 1) that is not justified and that sits in tension with the paper's own task-dependence property. The formal definition of multiplicity is also vacuous as written. If these foundational issues are fixed, the paper could serve as a useful agenda-setting contribution; in its current form, the argument is not fully rigorous.

major comments (3)
  1. [Section 2, Definition] The formal definition of multiplicity is stated as |{(x,y) in R}| >= 1, but this inequality is also satisfied by the standard one-to-one assumption (exactly one positive per query). As written, the definition cannot distinguish multiplicity from ordinary alignment and is therefore vacuous. It should be defined as the existence of at least one x or y with more than one valid counterpart (e.g., |{(x,y) in R}| > 1 for at least one x or y).
  2. [Section 2, Property 1] Property 1 asserts that if two positive pairs (x1,y1) and (x2,y2) have overlapping concepts, then the cross-pairs (x1,y2) and (x2,y1) are also positive. This is a transitivity assumption under concept overlap, and it is neither stated as an axiom nor defended. It fails in natural cases: an image of a sitting cat and a caption "a sleeping cat" share the concept "cat," yet they do not form a valid pair under a task requiring state consistency. Moreover, Property 3 (task-dependent alignment) explicitly allows the same pair to be positive for one task and negative for another, which is incompatible with an unconditional transitivity claim. Since Property 1 is used to convert intra-modal variability into unannotated cross-modal positives, the inevitability argument depends on it. Please either replace Property 1 with a weaker, task-relative property, or give precise conditions under which concept overlap suffices for positivity.
  3. [Section 3.1, Settings] The line "the total number of true positive relations is N - K + K^2" is only correct if K semantically equivalent images are each paired with all K semantically equivalent captions, forming a complete bipartite block. However, the preceding text only states "suppose x1 actually has K > 1 equally valid matches in the mini-batch." If only x1 has multiple matches, the total number of true positives is N + K - 1 (or N, depending on whether the other K - 1 annotated pairs are also counted), not N - K + K^2. Please clarify the intended assumption in the settings paragraph and correct the formula if a complete block is not assumed.
minor comments (5)
  1. [Section 3.1, near Eq. (1)] The phrase "N annotated positive pair" should be "N annotated positive pairs." Also, "pushing f(x1) and f(y1) closer" should read "pushing f(x1) and g(y1) closer," since f and g are the two modality encoders.
  2. [Reference [54]] Reference [54] lists "Anonymous authors" and "Under review." In a public arXiv preprint, this is unusual and should be corrected, or the entire reference should be anonymized consistently if necessary.
  3. [Reference [73]] Reference [73] contains an extra comma in the author list: "Wonjae Kim, Sanghyuk Chun, ..., , Yoohoon Kang" should be "... ..., Yoohoon Kang."
  4. [Figure 4 caption] The abbreviation "BT model" is used without definition. On first use, please spell out "Bradley-Terry model" as [64] does.
  5. [Section 3.2] The term "multimodal LLMs" is used without definition; consider spelling out or using "large multimodal models" on first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's empirical anchors are human-verified benchmarks, and its formal claims are conditional on explicit premises rather than derived from their own conclusions.

full rationale

The paper is a position paper whose argument is conditional and premise-based, not circular. Section 2 defines multiplicity via R and then introduces Properties 1-3 as explicit assumptions about how real-world multimodal data are generated (intra-modal variability, representational asymmetry, and task-dependent alignment). These premises are not defined in terms of the paper's conclusion, and the conclusion that multiplicity is inevitable is an inference from them, not a restatement. Section 3.1's gradient analysis is a conditional mathematical result: if a mini-batch contains K>1 unannotated positives, then the sparse contrastive loss induces an error term relative to the ideal loss; the derivation does not presuppose the conclusion, only the stated K>1 premise. The key quantitative evidence (8.5/17.9 in Property 1 and the roughly 4.4x figure in Section 4) comes from human-in-the-loop benchmarks, ECCV Caption and PCME, which are externally constructed and human-verified rather than fitted within this paper; the fact that those works are first-authored by the present first author does not make the citation circular, since the cited measurements are independent empirical evidence. The weakest step, Property 1's transitivity assumption that overlapping concepts imply valid cross-pairs, is an openly stated premise with real-world counterexamples possible; that is a correctness or calibration risk, not a circularity, because the paper does not claim to derive that premise from the target conclusion. Finally, the advocacy of probabilistic embeddings is a proposed research direction and is explicitly presented as one of several options, not as a forced consequence of the analysis. No equation or argument in the paper reduces its conclusion to its input by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are introduced in this position paper. The axioms list the structural assumptions the argument depends on, including the transitivity of semantic similarity and the existence of a task-independent ground-truth relation set.

assumptions (4)
  • domain assumption There exists a ground-truth relation set R of all valid cross-modal correspondences, independent of annotation or task.
    The definition of multiplicity in Section 2 presumes a well-defined R. The paper later admits alignment is task-dependent (Property 3), so a single R may not exist across tasks, yet the formalization uses a fixed R.
  • domain assumption Semantic similarity within a modality implies cross-modal matching: if two pairs share overlapping concepts, cross-pairings are also valid positives.
    Property 1 in Section 2 makes this transitivity assumption. It is the core mechanism that turns sparse annotation into false negatives. If this assumption fails, many cited FNs would not be true positives.
  • domain assumption Human annotator judgments in the cited studies (ECCV Caption, PCME) correctly identify valid correspondences.
    The paper's quantitative claims about the scale of false negatives (8.5/17.9, 4.4x) rely on human annotation from self-authored papers. No independent verification is provided.
  • standard math The softmax-based contrastive loss gradient is the relevant training mechanism.
    Section 3.1 uses standard softmax gradient calculations. These are correct but assume normalized embeddings and a specific loss form.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning." pith.science (2026). https://pith.science/paper/NDJB5T4G

@misc{pith2026250519614,
  author       = {Pith},
  title        = {Pith review of: Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NDJB5T4G}},
  note         = {Machine review of arXiv:2505.19614}
}
read the original abstract

Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities. Most current approaches are built on the assumption of a deterministic one-to-one alignment between modalities. However, this oversimplifies real-world multimodal relationships, where their nature is inherently many-to-many. The many-to-many property, or multiplicity, is not a side-effect of noise or annotation error, but an inevitable outcome of intra-modal variability, representational asymmetry, and task-dependent ambiguity in multimodal tasks. We argue that multiplicity is a fundamental bottleneck that affects all stages of the multimodal learning pipeline: from data construction to model training and evaluation benchmarks. By formalizing its causes and consequences, we demonstrate how ignoring multiplicity leads to training uncertainty, unreliable evaluation, and degraded dataset quality. This position paper calls for new research directions on multimodal learning, including multiplicity-aware learning frameworks and dataset construction and evaluation protocols.

Figures

Figures reproduced from arXiv: 2505.19614 by the authors.

Figure 1
Figure 1. How unimodal task and multimodal task are different? Unimodal tasks assume a fixed and pre-defined label set. Even though we add more instances in the dataset, the number of correspondences increases constantly, and the new instance does not affect to the existing instances. However, the correspondences in multimodal datasets, assuming one-to-one mapping, increase O(N) by adding one multimodal pair. number of annota… view at source ↗
Figure 2
Figure 2. How multiplicity occurs? The source of multiplicity in multimodal datasets is diverse. 2 Multiplicity: An inevitable and inherent challenge Definition. Let R ⊆ X × Y denote valid cross-modal relations between two modalities X and Y (e.g., vision-language [1], audio-visual [5]); Note that we assume two modalities for simplicity, but this definition can be easily extended to multiple modalities, such as vision-languag… view at source ↗
Figure 3
Figure 3. Multiplicity induces ambiguity. (a) If we have an ideal dataset consists of the full pairwise annotations, an input should correspond to multiple instances from the other modality. The current one-to-one paradigm cannot handle this. (b) In practice, we have sparsely annotated pairwise annotations: each input only corresponds to one instance. In this case, multiplicity introduces a new uncertainty, named matching amb… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Human preference vs. evaluation metrics under multiplicity. Chun et al. [56] asked human annotators to compare four retrieval scenarios: (A) only top-1 is wrong, (B) only top-1 is correct, (C) top-1 to top-5 are wrong, and (D) only top-5 is correct. By comparing them i…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 68 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021. 1, 3, 4, 5, 6, 8, 9

    2021

  2. [2]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational Conference on Machine Learning (ICML), pages 4904–4916. PMLR, 2021. 1, 8, 9

    work page 2021

  3. [3]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022. 1

    work page 2022

  4. [4]

    Visual instruction tuning.Advances in Neural Information Processing Systems (NeurIPS), 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in Neural Information Processing Systems (NeurIPS), 36:34892–34916, 2023. 1, 5, 6

    work page 2023

  5. [5]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. InICASSP, pages 1–5. IEEE, 2023. 1, 3

    work page 2023

  6. [6]

    Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022. 1, 3, 6

    arXiv 2022

  7. [7]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. InInternational Conference on Machine Learning (ICML), 2023. 1, 3

    work page 2023

  8. [8]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 1

    arXiv 2024

Show all 81 references
  1. [9]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection a...

    1956

  2. [10]

    Microsoft COCO captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 1, 3, 4, 6, 7, 8

    2015 arXiv

  3. [11]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge.International Journal of Computer Vision (IJCV...

  4. [12]

    Read, watch and scream! sound generation from text and video

    Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. InAAAI Conference on Artificial Intelligence, 2025. 3, 9

    2025

  5. [13]

    Are we done with ImageNet?arXiv preprint arXiv:2006.07159, 2020

    Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet?arXiv preprint arXiv:2006.07159, 2020. 3, 6, 8, 9

    2006 arXiv

  6. [14]

    Evaluating machine accuracy on imagenet

    Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on imagenet. InInternational Conference on Machine Learning (ICML), pages 8634–8644. PMLR, 2020. 3, 6, 8

    2020

  7. [15]

    Re- labeling imagenet: from single to multi-labels, from global to localized labels

    Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re- labeling imagenet: from single to multi-labels, from global to localized labels. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3, 6, 8

    2021

  8. [16]

    Models, reasoning and inference.Cambridge, UK: CambridgeUniversityPress, 19(2):3,

    Judea Pearl et al. Models, reasoning and inference.Cambridge, UK: CambridgeUniversityPress, 19(2):3,

  9. [17]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. InEuropean Conference on Computer Vision (ECCV), 2014. 3, 8 10

    2014

  10. [18]

    Probabilistic embeddings for cross-modal retrieval

    Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3, 6, 7, 9

    2021

  11. [19]

    Oxford university press, 1990

    Allan Paivio.Mental representations: A dual coding approach. Oxford university press, 1990. 4

    1990

  12. [20]

    Connecting vision and language with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. InEuropean Conference on Computer Vision (ECCV), 2020. 4, 8

    2020

  13. [21]

    Visualcomet: Reasoning about the dynamic context of a still image

    Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of a still image. InEuropean Conference on Computer Vision (ECCV), 2020. 4

    2020

  14. [22]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. InICASSP, pages 721–725. IEEE, 2020. 4

    2020

  15. [23]

    V oxceleb: a large-scale speaker identification dataset.arXiv preprint arXiv:1706.08612, 2017

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. V oxceleb: a large-scale speaker identification dataset.arXiv preprint arXiv:1706.08612, 2017. 4

    2017 arXiv

  16. [24]

    Visually indicated sounds

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2405–2413, 2016. 4

    2016

  17. [25]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. InAdvances in Neural Information Processing Systems (NeurIPS), pages 13–23, 2019. 4, 5

    2019

  18. [26]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. InInternational Conference on Machine Learning (ICML), 2021. 4, 5

    2021

  19. [27]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InInternational Conference on Computer Vision (ICCV), pages 11975–11986, 2023. 4, 5, 6

    2023

  20. [28]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558–3568, 2021. 4, 8

    2021

  21. [29]

    RedCaps: Web-curated image-text data created by the people, for the people

    Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. InNeurIPS Dataset and Benchmark (NeurIPS D&B), 2021. 4

    2021

  22. [30]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114, 2021

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114, 2021. 4

    2021 arXiv

  23. [31]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Information Processing Systems (NeurIPS), 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Lai...

    2022

  24. [32]

    Datacomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Systems (NeurIPS), 36,

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing System...

  25. [33]

    VSE++: Improving visual-semantic embeddings with hard negatives

    Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improving visual-semantic embeddings with hard negatives. InBritish Machine Vision Conference (BMVC), 2018. 5

    2018

  26. [34]

    Learning the best pooling strategy for visual semantic embedding

    Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15789–15798, 2021. 5

    2021

  27. [35]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems (NeurIPS), 35:17612–17625, 2022

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems (NeurIPS), 35:17612–17625, 2022. 5

    2022

  28. [36]

    Boosting contrastive self-supervised learning with false negative cancellation

    Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2785–2795, 2022. 5 11

    2022

  29. [37]

    Mafa: Managing false negatives for vision-language pre-training

    Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision-language pre-training. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27314–27324, 2024. 6

    2024

  30. [38]

    Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations (ICLR), 2022. 6

    2022

  31. [39]

    Improved probabilistic image-text representations

    Sanghyuk Chun. Improved probabilistic image-text representations. InInternational Conference on Learning Representations (ICLR), 2024. 6

    2024

  32. [40]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. InInternational Conference on Learning Representations (ICLR), 2018. 6

    2018

  33. [41]

    CutMix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. InInternational Conference on Computer Vision (ICCV), 2019. 6

    2019

  34. [42]

    Learning with noisy correspondence for cross-modal matching

    Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, hua wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems (NeurI...

    2021

  35. [43]

    Learning from noisy labels with deep neural networks: A survey.IEEE transactions on neural networks and learning systems, 34(11):8135–8153, 2022

    Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey.IEEE transactions on neural networks and learning systems, 34(11):8135–8153, 2022. 6

    2022

  36. [44]

    Learning from positive and unlabeled data: A survey.Machine Learning, 109(4):719–760, 2020

    Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey.Machine Learning, 109(4):719–760, 2020. 6

    2020

  37. [45]

    Polysemous visual-semantic embedding for cross-modal retrieval

    Yale Song and Mohammad Soleymani. Polysemous visual-semantic embedding for cross-modal retrieval. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1979–1988, 2019. 6, 9

    1979

  38. [46]

    Improving cross-modal retrieval with set of diverse embeddings

    Dongwon Kim, Namyup Kim, and Suha Kwak. Improving cross-modal retrieval with set of diverse embeddings. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23422–23431, 2023. 6

    2023

  39. [47]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017. 6

    2017 arXiv

  40. [48]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024. 6

    2024 arXiv

  41. [49]

    Probvlm: Proba- bilistic adapter for frozen vison-language models

    Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvlm: Proba- bilistic adapter for frozen vison-language models. InInternational Conference on Computer Vision (ICCV), pages 1899–1910, 2023. 6

    1910

  42. [50]

    Prototype-based aleatoric uncertainty quantification for cross-modal retrieval.Advances in Neural Information Processing Systems (NeurIPS), 36:24564–24585, 2023

    Hao Li, Jingkuan Song, Lianli Gao, Xiaosu Zhu, and Hengtao Shen. Prototype-based aleatoric uncertainty quantification for cross-modal retrieval.Advances in Neural Information Processing Systems (NeurIPS), 36:24564–24585, 2023. 6

    2023

  43. [51]

    Post-hoc probabilistic vision-language models.arXiv preprint arXiv:2412.06014, 2024

    Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, and Martin Trapp. Post-hoc probabilistic vision-language models.arXiv preprint arXiv:2412.06014, 2024. 6

    2024

  44. [52]

    Probabilistic language-image pre-training

    Sanghyuk Chun, Wonjae Kim, Song Park, and Sangdoo Yun. Probabilistic language-image pre-training. In International Conference on Learning Representations (ICLR), 2025. 6, 9

    2025

  45. [53]

    LongProLIP: A probabilistic vision-language model with long context text

    Sanghyuk Chun and Sangdoo Yun. LongProLIP: A probabilistic vision-language model with long context text. InICLR Workshop on Quantify Uncertainty and Hallucination in Foundation Models, 2025. 6

    2025

  46. [54]

    Seeing what you say: Expressive image generation from speech.Under review, 2025

    Anonymous authors. Seeing what you say: Expressive image generation from speech.Under review, 2025. 6

    2025

  47. [55]

    Flaws of imagenet, com- puter vision’s favorite dataset

    Nikita Kisel, Illia V olkov, Kateˇrina Hanzelková, Klara Janouskova, and Jiri Matas. Flaws of imagenet, com- puter vision’s favorite dataset. InICLR Blogposts 2025, 2025. https://d2jud02ci9yv69.cloudfront.net/2025- 04-28-imagenet-flaws-135/blog/imagenet-flaws/. 6 12

    2025

  48. [56]

    ECCV Caption: Correcting false negatives by collecting machine-and-human-verified image-caption associations for MS-COCO

    Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang Chang, and Seong Joon Oh. ECCV Caption: Correcting false negatives by collecting machine-and-human-verified image-caption associations for MS-COCO. InEuropean Conference on Computer Vision (ECCV), 2022. 7, 8

    2022

  49. [57]

    A metric learning reality check

    Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. InEuropean Conference on Computer Vision (ECCV), 2020. 7

    2020

  50. [58]

    Holistic evaluation of text-to-image models

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems (NeurIPS), 36:69981–70011, 2023. 7

    2023

  51. [59]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. 7

    2017

  52. [60]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InAssociation for Computational Linguistics (ACL), pages 311–318, 2002. 7

    2002

  53. [61]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004. 7

    2004

  54. [62]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. 7

    2005

  55. [63]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018. 7

    2018

  56. [64]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. 7

    1952

  57. [65]

    The Caltech-UCSD Birds-200-2011 Dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 8

    2011

  58. [66]

    On semantic similarity in video retrieval

    Michael Wray, Hazel Doughty, and Dima Damen. On semantic similarity in video retrieval. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3650–3660, 2021. 8

    2021

  59. [67]

    Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for MS-COCO

    Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for MS-COCO. InConference of the European Chapter of the Association for Computational Linguistics (EACL), 2021. 8

    2021

  60. [68]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InIEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2023

  61. [69]

    T-mars: Improving visual representations by circumventing text feature learning

    Pratyush Maini, Sachin Goyal, Zachary C Lipton, J Zico Kolter, and Aditi Raghunathan. T-mars: Improving visual representations by circumventing text feature learning. InInternational Conference on Learning Representations (ICLR), 2024. 8

    2024

  62. [70]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. InInternational Conference on Learning Representations (ICLR), 2024. 8

    2024

  63. [71]

    Text-only training for image captioning using noise- injected clip

    David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise- injected clip. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022. 8

    2022

  64. [72]

    I can’t believe there’s no images! learning visual tasks using only language supervision

    Sophia Gu, Christopher Clark, and Aniruddha Kembhavi. I can’t believe there’s no images! learning visual tasks using only language supervision. InInternational Conference on Computer Vision (ICCV), pages 2672–2683, 2023. 8

    2023

  65. [73]

    Decap: Decoding clip latents for zero-shot captioning via text-only training

    Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. InInternational Conference on Learning Representations (ICLR), 2023. 8

    2023

  66. [74]

    Language-only efficient training of zero-shot composed image retrieval

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, , Yoohoon Kang, and Sangdoo Yun. Language-only efficient training of zero-shot composed image retrieval. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 8 13

    2024

  67. [75]

    The devil is in the details: A deep dive into the rabbit hole of data filtering.arXiv preprint arXiv:2309.15954, 2023

    Haichao Yu, Yu Tian, Sateesh Kumar, Linjie Yang, and Heng Wang. The devil is in the details: A deep dive into the rabbit hole of data filtering.arXiv preprint arXiv:2309.15954, 2023. 9

    2023 arXiv

  68. [76]

    Icons: Influence consensus for vision-language data selection.arXiv preprint arXiv:2501.00654, 2024

    Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, and Olga Russakovsky. Icons: Influence consensus for vision-language data selection.arXiv preprint arXiv:2501.00654, 2024. 9

    2024

  69. [77]

    HYPE: Hyperbolic entailment filtering for underspecified images and texts

    Wonjae Kim, Sanghyuk Chun, Taekyung Kim, Dongyoon Han, and Sangdoo Yun. HYPE: Hyperbolic entailment filtering for underspecified images and texts. InEuropean Conference on Computer Vision (ECCV), 2024. 9

    2024

  70. [78]

    CompoDiff: Versatile composed image retrieval with latent diffusion.Transactions on Machine Learning Research (TMLR), 2024

    Geonmo Gu, Sanghyuk Chun, HeeJae Jun, Yoohoon Kang, Wonjae Kim, and Sangdoo Yun. CompoDiff: Versatile composed image retrieval with latent diffusion.Transactions on Machine Learning Research (TMLR), 2024. 9

    2024

  71. [79]

    Toward interactive regional understanding in vision- large language models

    Jungbeom Lee, Sanghyuk Chun, and Sangdoo Yun. Toward interactive regional understanding in vision- large language models. InAnnual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024. 9

    2024

  72. [80]

    Crepe: Can vision-language foundation models reason compositionally? InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10910–10921, 2023

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10910–10921, 2023. 9

    2023

  73. [81]

    Large-scale interactive object segmentation with human annotators

    Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11700–11709, 2019. 9 14

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.