Pith. sign in

REVIEW 4 major objections 5 minor 85 references

Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Contrastive pretraining can shed its 'myopic' one-short-text bias by giving each image multiple captions and one visual branch per caption, trained with a multi-to-multi loss — consistently beating one-to-multi on the same data.

desk verdict A useful multi-caption CLIP recipe, but the headline M2M-vs-O2M comparison never holds the encoder/fusion constant; add that control before believing the loss attribution. read the letter →

arxiv 2412.00440 v1 pith:KHERICSV submitted 2024-11-30 cs.CV

Haicheng Wang , Chen Ju , Weixiong Lin , Shuai Xiao , Mengting Chen , Yixuan Huang , Chang Liu , Mingshuai Yao
show 4 more authors
Jinsong Lan Ying Chen Qingwen Liu Yanfeng Wang
This is my paper · ORCID
classification cs.CV
keywords contrastivelanguage-imagepre-trainingmulti-to-multilearningmulti-branchimageencoderdiversecaptiongenerationimage-textretrievalzero-shotclassificationdensevisualunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard CLIP-style contrastive pretraining is 'myopic': because each web image is paired with a single short text and squeezed into one embedding vector, the model misses the many valid descriptions of an image and underperforms on long-text, fine-grained, and dense visual tasks. To fix this, the authors generate several captions per image with one captioning model using five different prompt 'spirits' (details, nouns, main object, background, style), then give the image encoder one branch per caption and train with a multi-to-multi contrastive loss that aligns each branch to its own caption rather than all captions to one vector. The paper reports that this holistic pipeline beats both the original CLIP and one-to-multi contrastive variants on the same data across image-text retrieval, zero-shot classification, captioning, and dense visual question-answering benchmarks. If the claim holds, the recipe is simple: diversify captions, dedicate a branch per caption type, and let contrastive pre-training decompose the image accordingly.

What carries the argument

The load-bearing machinery is the multi-to-multi contrastive loss together with a multi-branch image encoder. The encoder is modified in one of two cheap ways: initialize H extra class tokens that each attend to the image and output their own embedding, or extend the last three MLP layers into H parallel heads (about 3 percent parameter increase per head). The loss extends InfoNCE from one-to-one to one-to-multi: for each of M captions, it pulls the matching visual branch toward that caption and pushes other images' same-branch embeddings away; text-to-image and image-to-text directions are averaged symmetrically. The prompt set is equally central: five prompts (Details, Nouns, Main Object, Background, Style) steer one captioning model to produce texts that are genuinely different from each other, which is what gives the branches distinct semantics to learn.

What would settle it

Train the identical multi-to-multi pipeline but randomly permute which prompt type is assigned to which image branch across images. If retrieval and classification performance stays essentially the same, then the per-branch matching is not actually carrying the gains. A second check: measure average pairwise cosine similarity among the five generated captions on a held-out sample; if it approaches the similarity of captioning the same image twice with the same prompt, the 'diversity' premise fails.

Watch

Extended reading notes

Core claim

The central discovery claimed is that an image embedding does not have to be a single summary vector. By starting with a (image, multi-text) dataset where each image has M captions from different perspectives, and modifying the CLIP image encoder to output H parallel embeddings (either H initialized class tokens or H parallel MLP heads), the model can learn part-to-part alignment. The multi-to-multi contrastive loss treats each (visual branch, text) pair as a positive pair and pulls them together, so each branch learns a specific aspect of the image. The paper's experiments across ten benchmarks show consistent gains over the one-to-multi baseline on identical data, with the largest margins when the captions are made diverse through carefully designed prompts rather than through using multiple different captioning models.

Load-bearing premise

The five hand-written prompts must make one captioning model produce captions that are both diverse and individually accurate enough that each caption can be cleanly assigned to one image branch; if the captions are redundant, noisy, or inconsistently ordered across images, the one-to-one branch-to-caption matching in the loss becomes arbitrary and most of the reported gains would come from having extra training text rather than from the multi-to-multi mechanism.

Editorial extensions

If this is right

  • On the same multi-text training data, the multi-to-multi objective outperforms one-to-multi on short-text and long-text retrieval, zero-shot classification, image captioning, and dense visual question-answering benchmarks.
  • Five prompts aimed at different aspects (details, nouns, main object, background, style) produce more useful training signal than five captions from five different captioning models, and generally more captions help.
  • Because the visual branches are disentangled, a branch that hurts a particular task (for example, background details on object-centric classification) can be dropped at inference without retraining.
  • The extra image branches add only a small parameter overhead (about 3 percent per head for the MLP variant), and average fusion restores a single embedding at no extra retrieval cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editor would want to test the mechanism directly by randomizing which prompt type is assigned to which image branch across images; if performance barely moves, the gains come from extra training text rather than from part-to-part matching.
  • The paper shows caption-pair similarity dropping as caption count grows, but not per-caption quality; measuring whether each of the five captions is individually accurate and distinct from the others on a held-out sample would pin down the data premise.
  • The disentangled branches suggest task-specific inference without retraining, such as using only the 'Main Object' branch for object classification and the 'Long Text' branch for retrieval-heavy queries, and the paper's Table 7 supports the feasibility.
  • Naturally multi-text data, such as product listings with attribute descriptions, merchant text, and user reviews, would be a direct transfer domain for the same branch-per-text recipe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a holistic extension of CLIP in which each image is paired with multiple VL M-generated captions (from a single VLM with diverse prompts or from multiple VLMs) and trained with a new multi-to-multi (M2M) contrastive loss. The image encoder is modified to produce H visual embeddings, either through extra CLS tokens (ΨCLS) or parallel MLP branches (ΨMLP), and the loss in Eq. (5) aligns each visual branch with one caption. The authors compare M2M against vanilla CLIP and one-to-multi (O2M) contrastive baselines on short-text retrieval, long-text retrieval, zero-shot classification, captioning, and LLaVA-based dense evaluation, reporting consistent gains on CC3M and CC12M.

Significance. If the attribution were clean, the paper would offer a practical recipe for CLIP-style pre-training: generate diverse captions, dedicate one image branch to each caption type, and train with per-branch contrastive matching. The empirical sweep is broad, the two multi-branch implementations are parameter-efficient, and the flexible inference customization in Table 7 is a nice interpretability bonus. The paper is also honest about limited compute and about not exploring very large caption counts. However, the central claim that M2M learning, rather than the accompanying multi-branch architecture and fusion strategy, causes the gains is not currently established, because the O2M and M2M comparisons never hold the encoder and inference fusion fixed.

major comments (4)
  1. [§3.3.2–3.3.3, Tables 1–5] The O2M-vs-M2M comparison confounds the loss with the encoder architecture and the inference fusion. In Eq. (4), O2M uses a single image embedding v, while M2M uses H embeddings from ΨCLS or ΨMLP and averages them at inference (Section 3.3.3, Table 6). The reported 1.5–7% gains attributed to Eq. (5) could therefore be explained by ensembling H learned features, by the added attention/MLP capacity of the multi-branch encoder, or by the AVG fusion strategy. Section 4.3 only compares ΨCLS with ΨMLP, both of which produce H embeddings, so it does not remove this confound. Please add an O2M arm that uses the same multi-branch encoder and the same inference fusion, or otherwise isolate the effect of the M2M loss from the effect of the multi-branch representation.
  2. [Tables 1–8] No seed variance or error bars are reported anywhere, and several central M2M-over-O2M gains are small (e.g., 0.3–3% in Table 1 and 1–4% in Table 2). Single-run comparisons at this scale may be within training noise. Please report mean and standard deviation over at least three seeds for the central comparisons, or justify why single seeds are adequate for these effect sizes.
  3. [Section 3.2, Remark; Section B.3] The paper's diversity measure is only the pairwise text similarity, not the semantic accuracy or distinctness of each prompted caption. If, for example, the 'Style' and 'Details' prompts produce overlapping captions, the one-to-one branch-to-caption assignment in Eq. (5) becomes arbitrary, and the reported gains could be due to more training text rather than to true part-to-part matching. Please add a human or automatic evaluation that verifies each prompt reliably produces captions with the intended semantic dimension.
  4. [Section 4.1; Appendix B.3; code URL] Reproducibility is incomplete for a data-centric method: the generated (image, multi-texts) captions are not released, and the GitHub link has no commit hash. The captioning pipeline (prompt set, filtering thresholds, and VLM versions) is described only in high-level terms, yet the captions are the core new data. Please release the captions for CC3M/CC12M or, at minimum, a documented sample with exact prompts and filtering code.
minor comments (5)
  1. [Section 3] The section header numbering is duplicated: 'Sec 3.2 reports image-text holistic alignment data' and 'Sec 3.2 details multi-to-multi contrast' should refer to Sections 3.2 and 3.3, respectively.
  2. [Eq. (5)] The notation in Eq. (5) should explicitly define v_{i,j} and t_{i,j}; currently the reader must infer that i indexes the batch and j indexes the M text branches, and the capitalization of K and M in the sums is inconsistent with Eq. (4).
  3. [Table 4] Table 4's header is confusing: the 'Training Text' column appears to mix the training dataset name and the number of captions without a clear sub-header. Please restructure the table so each row's training data and caption count are explicit.
  4. [Table 2 and Section 4.2] The dataset name is written inconsistently as 'Sharegpt4-5K' in Table 2 and 'Sharegpt4v-5K' in the text; use one name throughout.
  5. [Throughout] There are multiple typographical and spacing issues, including 'constrastive' in several places and 'LLaV A' instead of 'LLaVA'; a careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central M2M-versus-O2M comparison is empirical and not forced by construction; multi-branch encoding is an architectural change, not a relabeled fit.

full rationale

The paper's derivation chain is an empirical pre-training comparison, not a mathematical derivation with fitted constants relabeled as predictions. The central claim, that M2M (Eq. 5) outperforms O2M (Eq. 4) on identical (image, multi-text) data, is a direct experimental comparison across external benchmarks (Tables 1-5), and the two losses are genuinely different objectives: Eq. (5) with M=H assigns each visual branch to one caption type rather than collapsing M texts onto a single image embedding as in Eq. (4). No parameter is fitted to the benchmark targets and then reported as a prediction. The multi-branch image encoder (ΨCLS/ΨMLP) is an architectural modification, and while an O2M arm with the same multi-branch encoder is not reported, that is a possible experimental confound, not circularity by the paper's own definitions. The many self-citations in Related Work are contextual, and none is invoked as the load-bearing justification for the M2M result; the O2M baselines are external works [18, 47, 83] reproduced under the same data. The Appendix D limitation about not exploring large M or Laion400M is honest and does not reveal a circular step. Under the hard rules, no claim reduces by construction to its input, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities or external mechanisms are introduced; the multi-branch embeddings are internal model parameters, not invented entities. The free parameters are the hand-chosen caption count, prompt set, and fusion strategy.

free parameters (3)
  • caption count M and branch count H = 5 (with 4-of-5 at inference for some tasks)
    Chosen by hand and ablation in Table 8; the paper finds 5 captions works, and even drops the background branch post hoc to improve several benchmarks.
  • manual prompt set = Details, Nouns, Main Object, Background, Style (Table 9)
    The five prompts are hand-designed to enforce text diversity; there is no optimization, but performance depends on this exact choice.
  • inference fusion strategy = Average (chosen in Table 6)
    Max, norm-max, and average are compared and average is selected; this choice affects all retrieval and classification numbers.
assumptions (4)
  • domain assumption Multiple diverse captions generated by a single VLM under hand-written prompts are complementary and together describe the full image.
    Central to Section 3.2 and Eq. (2); if captions are redundant or noisy, the 'holism' premise fails.
  • domain assumption A dedicated visual branch can specialize to a caption type under the M2M loss, producing disentangled embeddings.
    Assumed in Section 3.3.3 and Eq. (5); the paper provides attention visualizations, but no formal guarantee that branch j's alignment to text j is exclusive.
  • standard math Standard CLIP assumptions hold: contrastive loss over a batch of independent noisy pairs learns useful cross-modal alignment.
    Inherited from Radford et al. [57] and used throughout Section 3.3.
  • domain assumption The evaluation sets are not contaminated by the captioners' pretraining data.
    The paper states testing sets are standardized, but does not analyze overlap between InternVL2/LLaVA/QwenVL pretraining data and Fashion200K, Sharegpt4V, ImageNet, or COCO images and captions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training." pith.science (2026). https://pith.science/paper/KHERICSV

@misc{pith2026241200440,
  author       = {Pith},
  title        = {Pith review of: Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHERICSV}},
  note         = {Machine review of arXiv:2412.00440}
}
read the original abstract

In rapidly evolving field of vision-language models (VLMs), contrastive language-image pre-training (CLIP) has made significant strides, becoming foundation for various downstream tasks. However, relying on one-to-one (image, text) contrastive paradigm to learn alignment from large-scale messy web data, CLIP faces a serious myopic dilemma, resulting in biases towards monotonous short texts and shallow visual expressivity. To overcome these issues, this paper advances CLIP into one novel holistic paradigm, by updating both diverse data and alignment optimization. To obtain colorful data with low cost, we use image-to-text captioning to generate multi-texts for each image, from multiple perspectives, granularities, and hierarchies. Two gadgets are proposed to encourage textual diversity. To match such (image, multi-texts) pairs, we modify the CLIP image encoder into multi-branch, and propose multi-to-multi contrastive optimization for image-text part-to-part matching. As a result, diverse visual embeddings are learned for each image, bringing good interpretability and generalization. Extensive experiments and ablations across over ten benchmarks indicate that our holistic CLIP significantly outperforms existing myopic CLIP, including image-text retrieval, open-vocabulary classification, and dense visual tasks.

Figures

Figures reproduced from arXiv: 2412.00440 by the authors.

Figure 1
Figure 1. Myopia. OpenAI’s CLIP [57] uses crude (image, text) web data for one-to-one contrastive alignment, causing serious myopia, i.e., bias to monotonous short texts and shallow visual expressivity. Holism. We advance one holistic CLIP paradigm, by updating colorful (image, multi-texts) data from diverse views, levels; and designing multi-to-multi constrastive learning for image-text part-to-part matching. Abstract In rap… view at source ↗
Figure 2
Figure 2. Pipeline Overview of Holistic CLIP. To evolve data from monotonous (image, text) pairs to colorful (image, multi-texts) pairs, we use powerful VLMs for captioning from multiple views, levels, and granularities. Diverse prompts are defined to encourage diversity. We then modify the CLIP image encoder into multi-branch, and optimize by multi-to-multi constrastive learning for part-to-part matching. During inference, f… view at source ↗
Figure 3
Figure 3. Attention Visualization of Holistic CLIP’s Vision. Vi￾sion is naturally decomposed by aligning with various texts. to further dissect the efficacy of text number. As the model is exposed to an increasing number of descriptive texts with differences, its performance continues to improve. By ex￾tending texts from four to five adding background descrip￾tions, we strengthen the model’s semantic representation. However, … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Architecture Overview of Holistic CLIP. To generate H image features, we leverage two different structures: ΨCLS and ΨMLP. Then we match H image features with M text features. Normally H = M and we apply one-to-one matching [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Examples of (Image, Multi-Texts) Data from Multi-VLMs. C. Visualizations Here we offer more visualizations on the attention distribu￾tion among different image class tokens in ΨMLP. In Fig￾ure 9, we demonstrate that our M2M contrastive learning paradigm allows to produ…
Figure 6
Figure 6. Figure 6: Examples of (Image, Multi-Texts) Data from Multi-VLMs. Background: The background of the image features a stunning, vibrant building standing out against a clear sky with a gradient of colors transitioning from blue to warm hues near the horizon, suggesting either sunr…
Figure 7
Figure 7. Figure 7: Examples of (Image, Multi-Texts) Data from Multi-Prompts. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Examples of (Image, Multi-Texts) Data from Multi-Prompts. DetailMain Object Background [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Visualization of Attention Maps Among Different Visual Class Tokens. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 54 canonical work pages

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 5, 14

    arXiv 2023

  2. [2]

    Opental: Towards open set temporal action localization

    Wentao Bao, Qi Yu, and Yu Kong. Opental: Towards open set temporal action localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ,

  3. [3]

    Is clip the main roadblock for fine-grained open- world perception? arXiv preprint arXiv:2404.03539, 2024

    Lorenzo Bianchi, Fabio Carrara, Nicola Messina, and Fab- rizio Falchi. Is clip the main roadblock for fine-grained open- world perception? arXiv preprint arXiv:2404.03539, 2024. 2, 3

    arXiv 2024

  4. [4]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021. 5, 13, 14

    2021

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 6

    arXiv 2023

  6. [6]

    Wear-any-way: Manip- ulable virtual try-on via sparse correspondence alignment

    Mengting Chen, Xi Chen, Zhonghua Zhai, Chen Ju, Xuewen Hong, Jinsong Lan, and Shuai Xiao. Wear-any-way: Manip- ulable virtual try-on via sparse correspondence alignment. arXiv preprint arXiv:2403.12965, 2024. 2

    arXiv 2024

  7. [7]

    Enhanc- ing cross-domain click-through rate prediction via explicit feature augmentation

    Xu Chen, Zida Cheng, Jiangchao Yao, Chen Ju, Weilin Huang, Jinsong Lan, Xiaoyi Zeng, and Shuai Xiao. Enhanc- ing cross-domain click-through rate prediction via explicit feature augmentation. In International World Wide Web Con- ference, 2024. 2

    2024

  8. [8]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. InProceedings of the European Conference on Computer Vision . Springer,

Show all 85 references
  1. [9]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE Conference on Computer Vis...

    2024

  2. [10]

    Vindlu: A recipe for effec- tive video-and-language pretraining

    Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: A recipe for effec- tive video-and-language pretraining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2023. 2

    2023

  3. [11]

    Denoiser: Rethinking the robustness for open-vocabulary action recognition

    Haozhe Cheng, Cheng Ju, Haicheng Wang, Jinxiang Liu, Mengting Chen, Qiang Hu, Xiaoyun Zhang, and Yanfeng Wang. Denoiser: Rethinking the robustness for open-vocabulary action recognition. arXiv preprint arXiv:2404.14890, 2024. 3

    2024 arXiv

  4. [13]

    Mixer: Image to multi-modal re- trieval learning for industrial application

    Zida Cheng, Shuai Xiao, Zhonghua Zhai, Xiaoyi Zeng, and Weilin Huang. Mixer: Image to multi-modal re- trieval learning for industrial application. arXiv preprint arXiv:2305.03972, 2023. 2

    2023 arXiv

  5. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2009. 5

    2009

  6. [15]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In Pro- ceedings of the International Conference on Machine Learn- in...

    2022

  7. [16]

    Learning to prompt for open-vocabulary ob- ject detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary ob- ject detection with vision-language model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 3

    2022

  8. [17]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.11691, 2024. 6, 7

    2024 arXiv

  9. [18]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Sys- tems, 2024. 2, 4, 5, 6, 7

    2024

  10. [19]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 6

    2023 arXiv

  11. [20]

    Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2024

  12. [21]

    Au- tomatic spatially-aware fashion concept discovery

    Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Au- tomatic spatially-aware fashion concept discovery. In Pro- 9 ceedings of the International Conference on Computer Vi- sion, 2017. 6

    2017

  13. [22]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020. 2

    2020

  14. [23]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 2020. 2

    2020

  15. [24]

    Ranking info noise con- trastive estimation: Boosting contrastive learning via ranked positives

    David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking info noise con- trastive estimation: Boosting contrastive learning via ranked positives. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, 2022. 2

    2022

  16. [25]

    Froster: Frozen clip is a strong teacher for open-vocabulary action recognition

    Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is a strong teacher for open-vocabulary action recognition. arXiv preprint arXiv:2402.03241, 2024. 3

    2024 arXiv

  17. [26]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Lear...

    2021

  18. [27]

    Point-level temporal action localization: Bridg- ing fully-supervised proposals to weakly-supervised losses

    Chen Ju, Peisen Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. Point-level temporal action localization: Bridg- ing fully-supervised proposals to weakly-supervised losses. arXiv preprint arXiv:2012.08236, 2020. 3

    2012 arXiv

  19. [28]

    Divide and conquer for single-frame temporal action localization

    Chen Ju, Peisen Zhao, Siheng Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Divide and conquer for single-frame temporal action localization. In Proceedings of the Interna- tional Conference on Computer Vision, 2021. 3

    2021

  20. [29]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In Proceedings of the European Conference on Computer Vision. Springer, 2022. 3

    2022

  21. [30]

    Adaptive mutual super- vision for weakly-supervised temporal action localization

    Chen Ju, Peisen Zhao, Siheng Chen, Ya Zhang, Xiaoyun Zhang, Yanfeng Wang, and Qi Tian. Adaptive mutual super- vision for weakly-supervised temporal action localization. IEEE Transactions on Multimedia, 2022. 3

    2022

  22. [31]

    Multi-modal prompting for low-shot temporal action localization

    Chen Ju, Zeqian Li, Peisen Zhao, Ya Zhang, Xiaopeng Zhang, Qi Tian, Yanfeng Wang, and Weidi Xie. Multi-modal prompting for low-shot temporal action localization. arXiv preprint arXiv:2303.11732, 2023. 3

    2023 arXiv

  23. [32]

    Turbo: Informativity- driven acceleration plug-in for vision-language models

    Chen Ju, Haicheng Wang, Zeqian Li, Xu Chen, Zhonghua Zhai, Weilin Huang, and Shuai Xiao. Turbo: Informativity- driven acceleration plug-in for vision-language models. arXiv preprint arXiv:2312.07408, 2023. 2

    2023 arXiv

  24. [33]

    Con- straint and union for partially-supervised temporal sentence grounding

    Chen Ju, Haicheng Wang, Jinxiang Liu, Chaofan Ma, Ya Zhang, Peisen Zhao, Jianlong Chang, and Qi Tian. Con- straint and union for partially-supervised temporal sentence grounding. arXiv preprint arXiv:2302.09850, 2023. 2, 3

    2023 arXiv

  25. [34]

    Distill- ing vision-language pre-training to collaborate with weakly- supervised temporal action localization

    Chen Ju, Kunhao Zheng, Jinxiang Liu, Peisen Zhao, Ya Zhang, Jianlong Chang, Qi Tian, and Yanfeng Wang. Distill- ing vision-language pre-training to collaborate with weakly- supervised temporal action localization. In Proceedings of the IEEE Conference on Computer Vision and Pa...

    2023

  26. [35]

    Turbo: Informativity-driven acceleration plug-in for vision-language large models

    Chen Ju, Haicheng Wang, Haozhe Cheng, Xu Chen, Zhonghua Zhai, Weilin Huang, Jinsong Lan, Shuai Xiao, and Bo Zheng. Turbo: Informativity-driven acceleration plug-in for vision-language large models. In Proceedings of the Eu- ropean Conference on Computer Vision. Springer, 2025. 2

    2025

  27. [36]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In Proceedings of the International Conference on Machine Learning. PMLR, 2021. 2

    2021

  28. [37]

    Veclip: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual-enriched captions. In Proceedings of the European Conference on Computer Vision . Springe...

    2025

  29. [38]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. 6

    2024

  30. [39]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceed- ings of the International Conference on Machine Learning . PMLR, 2022. 2

    2022

  31. [40]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 2

    2023 arXiv

  32. [41]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vi- sion. Springer, 2014. 5

    2014

  33. [42]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. 5, 6, 14

    2024

  34. [43]

    Ex- ploiting transformation invariance and equivariance for self- supervised sound localisation

    Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Ex- ploiting transformation invariance and equivariance for self- supervised sound localisation. In Proceedings of ACM Inter- national Conference on Multimedia, 2022. 2, 3

    2022

  35. [44]

    Audio-aware query-enhanced trans- former for audio-visual segmentation

    Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya Zhang. Audio-aware query-enhanced trans- former for audio-visual segmentation. arXiv preprint arXiv:2307.13236, 2023. 3

    2023 arXiv

  36. [45]

    Audio-visual segmentation via unlabeled frame exploitation

    Jinxiang Liu, Yikun Liu, Fei Zhang, Chen Ju, Ya Zhang, and Yanfeng Wang. Audio-visual segmentation via unlabeled frame exploitation. arXiv preprint arXiv:2403.11074, 2024. 2

    2024 arXiv

  37. [46]

    Annotation-free audio-visual segmentation

    Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, and Weidi Xie. Annotation-free audio-visual segmentation. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, 2024. 2

    2024

  38. [47]

    Mllms- augmented visual-language representation learning

    Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms- augmented visual-language representation learning. arXiv preprint arXiv:2311.18765, 2023. 4, 5, 6, 7

    2023 arXiv

  39. [48]

    Mmbench: Is your multi-modal model an all-around player? In Proceedings of the European Confer- ence on Computer Vision

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, 10 Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Proceedings of the European Confer- ence on Computer Vision. Springer, 2025. 6, 8

    2025

  40. [49]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, 2022. 6

    2022

  41. [50]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neu- rocomputing, 2022. 3

    2022

  42. [51]

    Diffusionseg: Adapting diffusion towards unsupervised object discovery

    Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arXiv:2303.09813, 2023. 2

    2023 arXiv

  43. [52]

    Open-vocabulary semantic segmenta- tion via attribute decomposition-aggregation

    Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. Open-vocabulary semantic segmenta- tion via attribute decomposition-aggregation. arXiv preprint arXiv:2309.00096, 2023. 3

    2023 arXiv

  44. [53]

    Open-vocabulary semantic segmentation via attribute decomposition-aggregation.Advances in Neural Information Processing Systems, 2024

    Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. Open-vocabulary semantic segmentation via attribute decomposition-aggregation.Advances in Neural Information Processing Systems, 2024. 3

    2024

  45. [54]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), 2019. 6

    2019

  46. [55]

    Clip- cap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021. 3, 6, 7

    2021 arXiv

  47. [56]

    Detailclip: Detail-oriented clip for fine-grained tasks

    Amin Karimi Monsefi, Kishore Prakash Sailaja, Ali Alilooee, Ser-Nam Lim, and Rajiv Ramnath. Detailclip: Detail-oriented clip for fine-grained tasks. arXiv preprint arXiv:2409.06809, 2024. 2, 3

    2024 arXiv

  48. [57]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proceedings of the International Conference ...

    2021

  49. [58]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  50. [59]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing , 2019. 13

    2019

  51. [60]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2022. 1, 2

    2022

  52. [61]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 15

    2021 arXiv

  53. [62]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In Proceedings of the European Conference on Com- puter Vision. Springer, 2022. 6

    2022

  54. [63]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  55. [64]

    Im- proving image captioning with better use of captions

    Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Im- proving image captioning with better use of captions. arXiv preprint arXiv:2006.11807, 2020. 2

    2006 arXiv

  56. [65]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, 2019. 6

    2019

  57. [66]

    Clip models are few-shot learners: Empirical studies on vqa and visual entailment

    Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. arXiv preprint arXiv:2203.07190,

    arXiv

  58. [67]

    Crepe: Learnable prompting with clip improves visual relationship prediction

    Rakshith Subramanyam, TS Jayram, Rushil Anirudh, and Ja- yaraman J Thiagarajan. Crepe: Learnable prompting with clip improves visual relationship prediction. arXiv preprint arXiv:2307.04838, 2023. 2

    2023 arXiv

  59. [68]

    Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks

    Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- t...

    2021

  60. [69]

    Towards open-vocabulary video instance segmentation

    Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards open-vocabulary video instance segmentation. In Proceed- ings of the International Conference on Computer Vision ,

  61. [70]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021. 3

    2021 arXiv

  62. [71]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. 2

    2021 arXiv

  63. [72]

    Multi-modal prototypes for open-set semantic segmentation

    Yuhuan Yang, Chaofan Ma, Chen Ju, Ya Zhang, and Yan- feng Wang. Multi-modal prototypes for open-set semantic segmentation. arXiv preprint arXiv:2307.02003, 2023. 3

    2023 arXiv

  64. [73]

    Multi-modal prototypes for open-world semantic segmentation

    Yuhuan Yang, Chaofan Ma, Chen Ju, Fei Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Multi-modal prototypes for open-world semantic segmentation. International Jour- nal of Computer Vision, 2024. 3

    2024

  65. [74]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and 11 Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In Proceedings of the International Conference on Learning Representations, 2022. 2

    2022

  66. [75]

    Un- supervised domain adaption via similarity-based prototypes for cross-modality segmentation

    Ziyu Ye, Chen Ju, Chaofan Ma, and Xiaoyun Zhang. Un- supervised domain adaption via similarity-based prototypes for cross-modality segmentation. In Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health: Third MICCAI ...

    2021

  67. [76]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the Association for Computational Linguistics, 2014. 5

    2014

  68. [77]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the International Conference on Machine Learning. PMLR, 2024. 6

    2024

  69. [78]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 2

    2021 arXiv

  70. [79]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 2

    2022

  71. [80]

    Long-clip: Unlocking the long-text capability of clip

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In Proceedings of the European Conference on Com- puter Vision. Springer, 2025. 2, 3

    2025

  72. [81]

    Ow-tal: Learning unknown human activities for open-world temporal action localization

    Yaru Zhang, Xiao-Yu Zhang, and Haichao Shi. Ow-tal: Learning unknown human activities for open-world temporal action localization. Pattern Recognition, 2022. 3

    2022

  73. [82]

    Bottom-up temporal action localization with mutual regularization

    Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian. Bottom-up temporal action localization with mutual regularization. In Proceedings of the European Con- ference on Computer Vision. Springer, 2020. 3

    2020

  74. [83]

    Dreamlip: Language- image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. In Proceedings of the European Conference on Computer Vision . Springer, 2025. 2, 4, 5, 6, 7

    2025

  75. [84]

    Non-contrastive learning meets language-image pre- training

    Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets language-image pre- training. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2023. 3

    2023

  76. [85]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. Inter- national Journal of Computer Vision, 2022. 2, 3

    2022

  77. [86]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2, 5, 14 12 In supplementary materials, we first provide more details about our...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.