Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Joint Generalized Cosine Similarity: A Novel Method for N-Modal Semantic Alignment Based on Contrastive Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Gram-determinant angle yields a single similarity score for any number of modality vectors and powers the GHA contrastive loss.

desk verdict A sound but over-scoped determinant-based n-way similarity with a useful contrastive loss; the 'arbitrary number of vectors' claim degenerates when n exceeds the embedding dimension, and the experiments are too thin to support the superiority claims. read the letter →

arxiv 2505.03532 v1 pith:2FAOTYDP submitted 2025-05-06 stat.AP

Yiqiao Chen , Zijian Huang This is my paper · ORCID
classification stat.AP
keywords JointGeneralizedCosineSimilarityGHALossmulti-modalcontrastivelearningsemanticalignmentGramdeterminantn-modalDerm7pt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most multi-modal contrastive learning is stuck with pairwise similarity: with three or more modalities, losses are computed between every pair and then summed. This paper proposes a single n-way similarity, the Joint Generalized Cosine Similarity (JGCS), defined as the cosine of the Gram hypervolume angle spanned by all modality vectors at once. It then builds a contrastive loss, GHA Loss, on this joint similarity and reports that on the three-modal Derm7pt dataset and on simulations it aligns modalities at least as well as pairwise baselines while running faster and tolerating added noise. If the method holds, multi-modal alignment with three, five, or twelve modalities no longer needs to be stitched together from pairwise terms.

What carries the argument

The load-bearing object is the Gram hypervolume angle (GHA): for n row vectors $f_i$ forming matrix $M$, the volume of the parallelepiped they span is $\sqrt{\det(MM^T)}$, and dividing that volume by the product of the vector lengths gives the sine of one generalized angle $\Theta$. JGCS is defined as $\cos\Theta$, and the loss plugs it into an InfoNCE-style contrastive term plus an angular equilibrium regularizer that penalizes variance among pairwise cosine similarities so no modality collapses onto the plane of another. The construction is rotation-invariant and permutation-symmetric, degenerates to ordinary cosine similarity for $n=2$, and requires non-negative embeddings (a ReLU final layer) to break the sign symmetry inherited from the squared determinant.

What would settle it

Run the GHA contrastive loss on synthetic n-tuples with $n=D+1$ and $D$-dimensional embeddings: every positive and negative tuple satisfies $\det(MM^T)=0$, so $\cos\Theta=1$ for all of them and the loss cannot separate positives from negatives. That directly tests whether the advertised 'arbitrary number of vectors' holds beyond the dimension.

Watch

Extended reading notes

Core claim

The central claim is that the Gram-determinant angle $\Theta$ given by $\Theta_{f_1,\dots,f_n}=\arcsin(\sqrt{\det(MM^T)}/\prod_{i=1}^n\|f_i\|_2)$, with $M=[f_1,\dots,f_n]^T$, is a valid and useful similarity for an n-tuple of vectors: $\cos\Theta=1$ for linearly dependent vectors and $\cos\Theta=0$ for pairwise orthogonal vectors. Replacing the pairwise cosine or dot product in a contrastive loss with $\cos\Theta$ yields the GHA Loss, which adds an angular equilibrium term that equalizes the pairwise cosine similarities inside each n-tuple. The paper claims this is the first similarity measurement that works for an arbitrary number of vectors, and its experiments show the joint objective outperforming or matching pairwise summed InfoNCE baselines across model configurations on Derm7pt while visibly clustering the three modalities of each sample in two-dimensional projections.

Load-bearing premise

The method silently requires that the number of vectors not exceed the embedding dimension; if there are more vectors than dimensions, every n-tuple has the same Gram-determinant volume and the similarity carries no information.

Editorial extensions

If this is right

  • Three or more modalities can be trained with one joint contrastive term instead of $\binom{n}{2}$ pairwise terms, removing hand-chosen weights for aggregated losses.
  • GHA Loss computation grows roughly linearly with the number of modalities, so high-modality pretraining avoids the exponential pairwise blow-up measured in the paper.
  • The same $\cos\Theta$ can serve outside contrastive learning as a general n-way similarity for clustering, retrieval, or diversity scoring of feature sets.
  • Because the angle is invariant to rotations, the alignment objective no longer depends on arbitrary coordinate choices in the shared embedding space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper advertises an arbitrary number of vectors, but the formula only discriminates when n does not exceed the embedding dimension D; at n > D every n-tuple has determinant zero and $\cos\Theta=1$, so a projection or higher-dimensional embedding is required before the method can be used on the 12-modal datasets cited in the introduction.
  • Replacing the ReLU positivity constraint with a reference axis or a signed angle would make JGCS usable in domains where non-negative embeddings are unnatural.
  • The angular equilibrium term is a diversity regularizer in disguise; it could be lifted out of contrastive learning into ensemble and feature-union problems where redundant signal is penalized.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multi-vector similarity measure, the Joint Generalized Cosine Similarity (JGCS), defined through the Gram-hypervolume angle between n feature vectors, together with a contrastive loss (GHA Loss) built on it. The authors claim this is the first similarity method capable of handling an arbitrary number of vectors, that the associated loss outperforms pairwise contrastive baselines for n-modal semantic alignment, and that it is noise-robust, computationally efficient, and scalable. The method is derived analytically in Section IV-A, and experiments are reported on the Derm7pt three-modal dataset and on simulated data in Section V.

Significance. If the method is restricted to the regime n ≤ D and the ReLU non-negativity constraint is accepted, the construction is a mathematically valid generalization of cosine similarity to n vectors: the Gram-determinant angle is rotation-invariant, permutation-symmetric, and reduces to the planar angle for n = 2. The computational-efficiency comparison in Figure 6 is a useful concrete observation, and the idea of replacing pairwise losses with a single n-way contrastive term is well motivated. However, the paper's headline claim of handling an arbitrary number of vectors is not correct as stated, because the Gram determinant vanishes whenever the number of vectors exceeds the embedding dimension; the empirical evidence for superiority over pairwise baselines is also incomplete, as Table I contains only single runs without error bars and the simulated contrastive experiment trains and evaluates on the same 4,000 samples. The central mathematical identity is sound, but the paper needs substantial clarification and additional experiments before its claims can be accepted.

major comments (4)
  1. [§I, §IV-A, Eq. (5)] The central claim that JGCS can handle an arbitrary number of vectors is false for n > D. For M ∈ R^{n×D}, the Gram matrix MM^T has rank at most D, so when n > D, det(MM^T) = 0, Eq. (5) gives Θ = 0, and cos Θ = 1 for every n-tuple. Consequently, in Eq. (7) all positive and negative logits equal exp(1/τ), the GHA loss is constant, and no gradient signal reaches the encoders. The paper mentions the condition n ≤ D only in the extremal property in §IV-A, not as a general constraint on the method, even though the introduction motivates the approach with 8-, 10-, and 12-modal datasets. Please either state the n ≤ D limitation prominently and provide a fallback for n > D (for example, projecting the n vectors into a subspace of dimension D' < n before computing the determinant), or remove the 'arbitrary number of vectors' claim from the abstract, introduction, and conclusion.
  2. [§IV-A, Eq. (1)] Eq. (1) writes V = |det(M)| = sqrt(det(MM^T)) for a non-square matrix M ∈ R^{n×D}, but det(M) is undefined when M is not square. The correct expression is V = sqrt(det(MM^T)). This is not merely a typographical issue, because the erroneous equality obscures the rank condition behind the n > D collapse and should be corrected for mathematical rigor.
  3. [§IV-A, final paragraph; §IV-C, Eqs. (10)–(12)] The method requires all feature vectors to be nonnegative, which the paper imposes by adding a ReLU activation to the final layer. This is an architectural constraint whose cost is not analyzed: it restricts the feasible representation space, prevents the representation of negative semantic directions, and can introduce dead ReLU units during training. No experiment isolates the effect of this constraint or compares against alternatives such as using squared norms or orientation-corrected angles. Since the non-negativity assumption is a prerequisite for the angle to be meaningful, the paper should either justify it more carefully or ablate it empirically.
  4. [§V-B, Table I; §V-C] The empirical claim that GHA Loss achieves superior performance is not fully supported. In Table I, the GHA setting loses to the Dual setting in several configurations (for example, Res101+Gate: GHA 59.90 vs. Dual 63.02; Res50+Gate: GHA 63.54 vs. Dual 65.89), and all reported numbers appear to come from single runs without error bars or significance tests, so the statement in §I that the results 'consistently demonstrate the high efficiency of the proposed method' is too strong. In §V-C, the simulated contrastive experiment trains the encoder on the same 4,000 samples used for evaluation, which demonstrates training-set alignment but not generalization. Please report repeated runs with standard deviations or confidence intervals, perform a meaningful train/test split, and temper the claims to match the observed differences.
minor comments (5)
  1. [§V-B] The sentence 'The convergence analysis of the model is depicted in Figure 1' appears to reference the wrong figure; the relevant convergence plots are shown in Figure 5, while Figure 1 depicts the method comparison diagram.
  2. [§IV-A, item 3] The permutation-symmetry property states that swapping feature vectors 'only alters the sign of the Gram matrix determinant,' but the Gram determinant det(MM^T) is invariant under row permutations because the permutation contributes a squared sign; only det(M) changes sign. The conclusion that Θ is unaffected is correct, but the stated reason is not.
  3. [§II-A] The sentence 'Models such as OSCAR and ViLBERT place a greater emphasis on modality interaction...' is repeated verbatim within the same subsection; one occurrence should be removed.
  4. [§II-B] The sentence 'Contrastive learning was first applied in the field of computer science and subsequently extended to the domain of natural language analysis' is imprecise; contrastive ideas have deep roots in linguistics and psycholinguistics, and the sentence should be rephrased to describe the machine-learning history more carefully.
  5. [§V-A] The hyperparameters τ = 0.005 and λ = 1 are reported, but no sensitivity analysis or discussion of how these values were chosen is provided; this is especially relevant because temperature strongly affects contrastive learning behavior.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the JGCS similarity and GHA loss are defined from the Gram determinant rather than fitted to the claimed outcome.

full rationale

The paper's central chain is self-contained. JGCS is defined by Eq. (5) as the arcsine of the normalized Gram determinant, the three-modal factorization in Eqs. (3)-(4) is an algebraic identity, and the similarity cosTheta is taken from that angle. The GHA loss in Eqs. (7)-(9) is a direct application of this definition to an InfoNCE-style objective, with no parameter fitted to the results it is later said to predict. The Derm7pt experiments use the official train/validation/test partition, providing an external check. The simulated 4,000-sample alignment experiment trains and evaluates on the same samples, so it is only an in-sample optimization demonstration; the paper makes no out-of-sample predictive claim from it, so I do not count it as a circular step. The n>D degeneracy (det(MM^T)=0 for n>D in Eq. (5)) is a scope limitation on the 'arbitrary number of vectors' claim, not a circular reduction of the derivation to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central identity is a classical Gram-determinant relation; no constants are fitted to reproduce results. The method nevertheless depends on an unstated n<=D condition, a ReLU positivity constraint to resolve sign symmetry, and hand-chosen hyperparameters in the loss. Invented entities: none.

free parameters (3)
  • temperature tau = 0.005
    InfoNCE temperature selected by hand in §V-A; controls sharpness of the contrastive distribution and affects all reported results.
  • regularization coefficient lambda = 1
    Weight of the angular equilibrium loss chosen by hand in §V-A; no sensitivity analysis is reported.
  • number of negative n-tuples K = 7
    Set to 7 for both GHA and Dual in §V-A; no ablation is reported.
assumptions (4)
  • standard math det(MM^T) equals the squared hypervolume spanned by the n row vectors of M (Cauchy-Binet).
    Used in Eq (1) and Eq (5) to define the hypervolume V and the angle Theta.
  • domain assumption The number of modalities n does not exceed the embedding dimension D.
    Required for det(MM^T) to be nonzero; when n>D the proposed similarity always equals 1. This is not flagged as a limitation.
  • ad hoc to paper All final feature vectors are nonnegative because a ReLU activation is added to the last layer.
    Introduced in §IV-A to remove the sign symmetry where {a,b,c} and {-a,b,c} give identical GHA; this restricts the model and is not justified by task requirements.
  • domain assumption Negative n-tuples formed by sampling modalities from different dataset samples are valid negatives for the InfoNCE-style objective.
    Assumed in §IV-B and needed for L_C to be a meaningful contrastive loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Joint Generalized Cosine Similarity: A Novel Method for N-Modal Semantic Alignment Based on Contrastive Learning." pith.science (2026). https://pith.science/paper/2FAOTYDP

@misc{pith2026250503532,
  author       = {Pith},
  title        = {Pith review of: Joint Generalized Cosine Similarity: A Novel Method for N-Modal Semantic Alignment Based on Contrastive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FAOTYDP}},
  note         = {Machine review of arXiv:2505.03532}
}
read the original abstract

Alignment remains a crucial task in multi-modal deep learning, and contrastive learning has been widely applied in this field. However, when there are more than two modalities, existing methods typically calculate pairwise loss function and aggregate them into a composite loss function for the optimization of model parameters. This limitation mainly stems from the drawbacks of traditional similarity measurement method (i.e. they can only calculate the similarity between two vectors). To address this issue, we propose a novel similarity measurement method: the Joint Generalized Cosine Similarity (JGCS). Unlike traditional pairwise methods (e.g., dot product or cosine similarity), JGCS centers around the angle derived from the Gram determinant. To the best of our knowledge, this is the first similarity measurement method capable of handling tasks involving an arbitrary number of vectors. Based on this, we introduce the corresponding contrastive learning loss function , GHA Loss, and the new inter-modal contrastive learning paradigm. Additionally, comprehensive experiments conducted on the Derm7pt dataset and simulated datasets demonstrate that our method achieves superior performance while exhibiting remarkable advantages such as noise robustness, computational efficiency, and scalability. Finally, it is worth mentioning that the Joint Generalized Cosine Similarity proposed by us can not only be applied in contrastive learning, but also be easily extended to other domains.

Figures

Figures reproduced from arXiv: 2505.03532 by the authors.

Figure 1
Figure 1. A comparison of the adjustment of the inter-modal distances in the embedding space between the GHA Loss and the traditional pairwise-calculated [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of the three modalities in the Derm7pt dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The radar visualization graphs of eight baseline models in terms of six evaluation indicators. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The number of times each of the three methods outperforms others in terms of the six indicators. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The convergence performance of the Res101+Concat model. The train cls loss represents the classification loss value calculated by the Focal loss, [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: A comparison of the computational efficiency of GHA Loss and Dual Loss under different numbers of negative samples and different numbers of [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The visualization of the Gram hypervolume angle (GHA). [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Two-dimensional embedding diagrams drawn by the t-SNE algorithm before and after the model uses the GHA Loss for inter-modal contrasive [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The error distribution of the joint generalized cosine similarity [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 20 canonical work pages

  1. [37]

    Con- trastive learning of medical visual representations from paired images and text,

    Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Con- trastive learning of medical visual representations from paired images and text,” in Machine learning for healthcare conference. PMLR, 2022, pp. 2–25

    work page 2022

  2. [1]

    Multi-modal sentiment analysis based on image and text fusion based on cross-attention mechanism,

    H. Li, Y . Lu, and H. Zhu, “Multi-modal sentiment analysis based on image and text fusion based on cross-attention mechanism,” Electronics, vol. 13, no. 11, p. 2069, 2024

    work page 2024

  3. [2]

    An image-text sentiment analysis method using multi-channel multi-modal joint learning,

    L. Gong, X. He, and J. Yang, “An image-text sentiment analysis method using multi-channel multi-modal joint learning,” Applied Artificial In- telligence, vol. 38, no. 1, p. 2371712, 2024

    work page 2024

  4. [3]

    Unified cross-modal attention: robust audio-visual speech recognition and beyond,

    J. Li, C. Li, Y . Wu, and Y . Qian, “Unified cross-modal attention: robust audio-visual speech recognition and beyond,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 32, pp. 1941–1953, 2024

    work page 1941

  5. [4]

    Catnet: Cross-modal fusion for audio–visual speech recognition,

    X. Wang, J. Mi, B. Li, Y . Zhao, and J. Meng, “Catnet: Cross-modal fusion for audio–visual speech recognition,” Pattern Recognition Letters, vol. 178, pp. 216–222, 2024

    work page 2024

  6. [5]

    Medm2g: Unifying medical multi-modal generation via cross-guided diffusion with visual invariant,

    C. Zhan, Y . Lin, G. Wang, H. Wang, and J. Wu, “Medm2g: Unifying medical multi-modal generation via cross-guided diffusion with visual invariant,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 11 502–11 512

    work page 2024

  7. [6]

    A comprehensive review on synergy of multi-modal data and ai technologies in medical diagnosis,

    X. Xu, J. Li, Z. Zhu, L. Zhao, H. Wang, C. Song, Y . Chen, Q. Zhao, J. Yang, and Y . Pei, “A comprehensive review on synergy of multi-modal data and ai technologies in medical diagnosis,” Bioengineering, vol. 11, no. 3, p. 219, 2024

    work page 2024

  8. [7]

    M3d: Advancing 3d medical image analysis with multi-modal large language models,

    F. Bai, Y . Du, T. Huang, M. Q.-H. Meng, and B. Zhao, “M3d: Advancing 3d medical image analysis with multi-modal large language models,” arXiv preprint arXiv:2404.00578 , 2024

    arXiv 2024

Show all 41 references
  1. [8]

    A review on methods and applications in multimodal deep learning,

    S. Jabeen, X. Li, M. S. Amin, O. Bourahla, S. Li, and A. Jabbar, “A review on methods and applications in multimodal deep learning,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 2s, pp. 1–41, 2023

    2023

  2. [9]

    Bimac: Bidirectional multimodal alignment in contrastive learning,

    M. Zareapoor, P. Shamsolmoali, and Y . Lu, “Bimac: Bidirectional multimodal alignment in contrastive learning,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 39, no. 21, 2025, pp. 22 290–22 298

    2025

  3. [10]

    What to align in multimodal contrastive learning?

    B. Dufumier, J. Castillo-Navarro, D. Tuia, and J.-P. Thiran, “What to align in multimodal contrastive learning?” arXiv preprint arXiv:2409.07402, 2024

    2024 arXiv

  4. [11]

    Variance alignment score: A simple but tough-to-beat data selection method for multimodal contrastive learning,

    Y . Wang, Y . Chen, W. Yan, K. Jamieson, and S. S. Du, “Variance alignment score: A simple but tough-to-beat data selection method for multimodal contrastive learning,” arXiv preprint arXiv:2402.02055 , 2024

    2024 arXiv

  5. [12]

    Contrastalign: Toward robust bev feature alignment via contrastive learning for multi-modal 3d object detection,

    Z. Song, F. Jia, H. Pan, Y . Luo, C. Jia, G. Zhang, L. Liu, Y . Ji, L. Yang, and L. Wang, “Contrastalign: Toward robust bev feature alignment via contrastive learning for multi-modal 3d object detection,” arXiv preprint arXiv:2405.16873, 2024

    2024 arXiv

  6. [13]

    Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing,

    J. Yang, H. Huang, Y . Zhou, X. Chen, Y . Xu, S. Yuan, H. Zou, C. X. Lu, and L. Xie, “Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing,” Advances in Neural Information Processing Systems, vol. 36, pp. 18 756–18 768, 2023

    2023

  7. [14]

    M5product: Self-harmonized contrastive learning for e-commercial multi-modal pretraining,

    X. Dong, X. Zhan, Y . Wu, Y . Wei, M. C. Kampffmeyer, X. Wei, M. Lu, Y . Wang, and X. Liang, “M5product: Self-harmonized contrastive learning for e-commercial multi-modal pretraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, ...

    2022

  8. [15]

    Humman: Multi-modal 4d human dataset for versatile sensing and modeling,

    Z. Cai, D. Ren, A. Zeng, Z. Lin, T. Yu, W. Wang, X. Fan, Y . Gao, Y . Yu, L. Pan et al. , “Humman: Multi-modal 4d human dataset for versatile sensing and modeling,” in European Conference on Computer Vision . Springer, 2022, pp. 557–577

    2022

  9. [16]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine,

    Y . Xie, C. Zhou, L. Gao, J. Wu, X. Li, H.-Y . Zhou, S. Liu, L. Xing, J. Zou, C. Xie et al. , “Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine,” arXiv preprint arXiv:2408.02900, 2024

    2024 arXiv

  10. [17]

    Mmearth: Exploring multi-modal pretext tasks for geospatial representation learning,

    V . Nedungadi, A. Kariryaa, S. Oehmcke, S. Belongie, C. Igel, and N. Lang, “Mmearth: Exploring multi-modal pretext tasks for geospatial representation learning,” in European Conference on Computer Vision . Springer, 2024, pp. 164–182

    2024

  11. [18]

    Gen-jema: enhanced explainability using generative joint embedding multimodal alignment for monitoring directed energy deposition,

    J. Ferreira, R. Darabi, A. Sousa, F. Brueckner, L. P. Reis, A. Reis, J. M. R. Tavares, and J. Sousa, “Gen-jema: enhanced explainability using generative joint embedding multimodal alignment for monitoring directed energy deposition,” Journal of Intelligent Manufacturing , pp. ...

    2025

  12. [19]

    Vse++: Improv- ing visual-semantic embeddings with hard negatives,

    F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler, “Vse++: Improv- ing visual-semantic embeddings with hard negatives,” arXiv preprint arXiv:1707.05612, 2017

    2017 arXiv

  13. [20]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  14. [21]

    Eva-clip: Improved training techniques for clip at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “Eva-clip: Improved training techniques for clip at scale,” arXiv preprint arXiv:2303.15389 , 2023

    2023 arXiv

  15. [22]

    Long-clip: Unlocking the long-text capability of clip,

    B. Zhang, P. Zhang, X. Dong, Y . Zang, and J. Wang, “Long-clip: Unlocking the long-text capability of clip,” in European Conference on Computer Vision. Springer, 2024, pp. 310–325

    2024

  16. [23]

    Oscar: Object-semantics aligned pre-training for vision-language tasks,

    X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei et al., “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX

    2020

  17. [24]

    Springer, 2020, pp. 121–137

    2020

  18. [25]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural information processing systems , vol. 32, 2019

    2019

  19. [26]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in International conference on machine learning . PMLR, 2021, pp. 5583–5594

    2021

  20. [27]

    Multimodal contrastive learning via uni-modal coding and cross-modal prediction for multimodal sentiment analysis,

    R. Lin and H. Hu, “Multimodal contrastive learning via uni-modal coding and cross-modal prediction for multimodal sentiment analysis,” arXiv preprint arXiv:2210.14556 , 2022

    2022 arXiv

  21. [28]

    Triplet-based deep hashing network for cross-modal retrieval,

    C. Deng, Z. Chen, X. Liu, X. Gao, and D. Tao, “Triplet-based deep hashing network for cross-modal retrieval,” IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 3893–3903, 2018

    2018

  22. [29]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  23. [30]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738

    2020

  24. [31]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning . PmLR, 2020, pp. 1597–1607

    2020

  25. [32]

    Semantic alignment network for multi-modal emotion recognition,

    M. Hou, Z. Zhang, C. Liu, and G. Lu, “Semantic alignment network for multi-modal emotion recognition,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 9, pp. 5318–5329, 2023

    2023

  26. [33]

    X2ct-clip: Enable multi- abnormality detection in computed tomography from chest radiography via tri-modal contrastive learning,

    J. You, Y . Gao, S. Kim, and C. Mcintosh, “X2ct-clip: Enable multi- abnormality detection in computed tomography from chest radiography via tri-modal contrastive learning,” arXiv preprint arXiv:2503.02162 , 2025

    2025 arXiv

  27. [34]

    Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis,

    S. Mai, Y . Zeng, S. Zheng, and H. Hu, “Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis,” IEEE Transactions on Affective Computing , vol. 14, no. 3, pp. 2276–2289, 2022

    2022

  28. [35]

    Tri-clt: Learning tri-modal representations with contrastive learning and transformer for multimodal sentiment recognition,

    Z. Yang, Z. Li, D. Zhu, and Y . Zhou, “Tri-clt: Learning tri-modal representations with contrastive learning and transformer for multimodal sentiment recognition,” Information Technology and Control , vol. 53, no. 1, pp. 206–219, 2024

    2024

  29. [36]

    Tricolo: Trimodal contrastive loss for text to shape retrieval,

    Y . Ruan, H.-H. Lee, Y . Zhang, K. Zhang, and A. X. Chang, “Tricolo: Trimodal contrastive loss for text to shape retrieval,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2024, pp. 5815–5825

    2024

  30. [38]

    Triangular contrastive learning on molecular graphs,

    M. Choi, W. Shin, Y . Lu, and S. Kim, “Triangular contrastive learning on molecular graphs,” arXiv preprint arXiv:2205.13279 , 2022

    2022 arXiv

  31. [39]

    Seven- point checklist and skin lesion classification using multitask multimodal neural nets,

    J. Kawahara, S. Daneshvar, G. Argenziano, and G. Hamarneh, “Seven- point checklist and skin lesion classification using multitask multimodal neural nets,” IEEE journal of biomedical and health informatics, vol. 23, no. 2, pp. 538–546, 2018

    2018

  32. [40]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  33. [41]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2980–2988

    2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.