Pith. sign in

REVIEW 4 major objections 5 minor 111 references

GeoMM: On Geodesic Perspective for Multi-modal Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing cosine distance with graph geodesic distance improves contrastive vision-language pre-training.

desk verdict Solid experiments but the geodesic loss as described cannot deliver the gradients the authors claim; the paper needs a detailed forward/backward or code before the results can be attributed to the mechanism. read the letter →

arxiv 2505.11216 v1 pith:EKXTSEJZ submitted 2025-05-16 cs.CV

Shibin Mei , Hang Wang , Bingbing Ni This is my paper · ORCID
classification cs.CV
keywords geodesicdistancemulti-modallearningcontrastivevision-languagepre-traininghierarchicalgraphshortestpathnegativesampleminingmomentumfeaturequeue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Geodesic distance measures separation along a curved space, and multimodal embeddings often live in such a space; the standard cosine similarity measures only a pairwise angle and can call two sentences similar when they mean different things. The paper's claim is that replacing cosine with geodesic distance, computed as shortest paths over a graph of the current sample pool, gives contrastive vision-language pre-training a more faithful notion of sample relatedness. It constructs a hierarchical graph over the momentum feature queue, connects points to cluster centers and cluster centers to their nearest neighbors, and inserts the shortest-path distance into the InfoNCE loss. Across image-text retrieval, VQA, NLVR2, visual entailment, and visual grounding, the geodesic versions of the base models consistently outperform their cosine counterparts, with fine-tuned COCO text retrieval R@1 rising from 78.0 to 79.3. A sympathetic reader should take the paper as establishing that the global topology of the feature pool is a usable and beneficial metric signal for multimodal contrastive learning.

What carries the argument

The load-bearing object is the hierarchical geodesic distance: a graph-distance oracle over the momentum feature queue, built as a two-level hierarchy in which each bottom-level point attaches to a K-means cluster center and cluster centers connect to their eight nearest neighbors; the geodesic distance between two samples is the trivial distance to the nearest cluster center plus the Floyd shortest path between centers, propagated up the hierarchy until the two points share a cluster. The distance is converted into a loss input by angle normalization: the accumulated angular path is truncated at $4\pi$, scaled to $[0,\pi]$, and mapped through cosine. This machinery makes distance depend on the entire sample pool rather than on a pair of points alone, and it does so at a computational cost that fits inside a large-queue contrastive training loop, with the graph fully rebuilt every 100 iterations and incrementally updated in between.

What would settle it

Train the same model with a control in which the geodesic distances are computed exactly as described but treated as fixed constants inside the loss, detached from the learning signal, and compare with the full method; if the detached control reproduces the reported R@1 gains, the improvement is not being driven by the geodesic geometry steering the encoders, and the central claim's mechanism is falsified. A second control that replaces the geodesic distances with distances along a randomly rewired graph with identical edge counts would isolate the topology content of the signal.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is a metric substitution: in the image-text contrastive loss, replace the cosine similarity between a query feature and the momentum feature queue with a geodesic distance defined on the queue's graph. The graph is built by connecting each point to its nearest neighbors; distances are shortest paths through this graph, so a query is close to a candidate only if the data manifold itself connects them through intermediate samples. This resolves a failure mode of pairwise metrics: two sentences can have near-identical word-level embeddings yet sit on different parts of the semantic manifold, and only a global, path-based distance reveals that. To keep the computation feasible at a queue size of 65,536, the paper proposes a two-layer hierarchical graph, with K-means clusters at the bottom and cluster centers connected in a k-nearest-neighbor graph at the top, using Floyd shortest paths between centers and incremental updates every 100 iterations. The accumulated path angle is truncated at $4\pi$, normalized to $[0,\pi]$, and mapped through the cosine function before the InfoNCE loss. The reported results across five downstream tasks support the claim that this substitution, not any architectural change, is what drives the improvement.

Load-bearing premise

The load-bearing premise is that the loss remains trainable: the paper computes the contrastive loss from geodesic distances that are outputs of a shortest-path algorithm on a discrete graph, and it never specifies how those distances change as the model's parameters change, so if they behave as fixed constants during training, the claimed improvement would not follow from the described mechanism.

Editorial extensions

If this is right

  • On the fine-tuned COCO benchmark, text retrieval R@1 rises from 78.0 for the cosine-based baseline to 79.3 for the geodesic version, and image retrieval rises from 61.2 to 62.5; Flickr30K moves similarly.
  • The metric swap transfers beyond the main baselines: zero-shot COCO retrieval improves when geodesic distance is added to other large-scale image-text contrastive models, and ImageNet linear classification improves for instance-discrimination self-supervised models.
  • Because the geodesic distance is global, it should reduce false negatives: samples that look alike in isolation but lie on different semantic paths remain far apart, so contrastive learning can mine harder, more reliable negatives.
  • The computational overhead is modest: for the base architecture, peak CUDA memory rises from 25.3G to 25.9G and per-100-iteration time from 255s to 268s.
  • The gains extend to reasoning tasks: VQA test-dev rises from 75.55 to 76.04, NLVR2 test-P from 80.96 to 81.63, and SNLI-VE val from 80.79 to 81.42 relative to the cosine-based baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One step beyond the paper: if geodesic distance helps because it encodes pool topology, the benefit should grow with queue size and sample-pool diversity; a testable extension would sweep queue sizes and measure whether the gap over cosine widens, whereas the paper fixes the queue at 65,536.
  • Another testable extension: because the graph is rebuilt only every 100 iterations and otherwise updated incrementally, the metric deliberately lags behind the moving encoders, and that slow-moving-target effect could itself be a regularizer independent of the geodesic geometry; a control that recomputes the graph every step or freezes it entirely would separate the two contributions.
  • The paper's theorems bound the number and scale of connected components per layer, so unreachable sample pairs, whose distance is set to infinity, are expected under the hierarchy; a reader could infer that part of the gains may come from this implicit filtering of easy negatives rather than from the path lengths themselves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GeoMM, a modification of contrastive vision-language pre-training that replaces cosine distance in the image-text contrastive (ITC) loss with a geodesic distance computed on a hierarchical graph of momentum features. The authors construct a two-level K-Means graph over a 65,536-entry feature queue, compute shortest-path distances between cluster centers with Floyd-Warshall, decompose out-of-graph query distances through the nearest bottom-level cluster center, and update the structure every 100 iterations. Experiments on ALBEF, TCL, and MAFA report consistent gains on image-text retrieval, VQA, NLVR2, SNLI-VE, and RefCOCO+, plus additional results on CLIP, FLIP, MOCOv2, and SwAV. The paper also states two theorems about connected-component counts in the hierarchical graph.

Significance. If the training mechanism were fully specified and the gains reproducible, the paper would make a useful contribution: it introduces a topological, graph-based distance into contrastive multimodal pre-training and demonstrates consistent improvements across multiple backbones and downstream tasks with modest computational overhead. The experimental coverage (zero-shot and fine-tuned retrieval, VQA, NLVR2, SNLI-VE, grounding, generalization to four other methods) is a genuine strength, and the efficiency analysis in Table 6 is useful. However, the significance is currently conditional: the central mechanism, gradient flow through geodesic distances, is not described, and several definitions are imprecise, so the paper does not yet establish why the reported gains should be attributed to the geodesic distance.

major comments (4)
  1. [Sec. 3.4-3.5, Algorithm 2] The central claim of the paper is that replacing cosine distance with the proposed geodesic distance in the contrastive loss accounts for the reported performance gains. As written, however, the algorithm cannot deliver gradients from that loss to the image and text encoders. The geodesic distances used in line 10 are produced by the Floyd shortest-path computation on a discrete graph (lines 3-5) and by the nearest-bottom-level-cluster-center decomposition of Eq. (3); both are piecewise-constant functions of the encoder features and have zero derivative almost everywhere. The paper does not specify a straight-through estimator, a soft cluster assignment, a detached-distance reweighting scheme, or any other differentiable surrogate, and the Numba/CUDA Floyd implementation described in Sec. 3.5 is not differentiable as written. If the distances are treated as constants when computing the InfoNCE loss in line 12, the logits are constants and the gradient with respect to theta is zero, so the described procedure cannot pull paired representations closer via the geodesic geometry. If instead one backpropagates through the discrete operations, the gradients vanish almost everywhere. In either case, the reported improvements are not explained by the described training mechanism, and the paper cannot be evaluated for correctness until this step is specified and, ideally, ablated (e.g., geodesic with straight-through gradient versus geodesic with detached distances versus cosine baseline).
  2. [Sec. 3.3, Eq. (2)] Equation (2) is self-referential and cannot be evaluated as printed: the right-hand side of the recursion for d(xi, C^k(xi)) contains d(xi, C^k(xi)) itself, so the distance from a point to its own cluster center is defined in terms of itself. The intended recursion presumably replaces that term with a distance to the next-level center (e.g., d(xi, C^{k+1}(xi))), but as printed the hierarchical geodesic distance computation is ill-defined. Please correct the formula and include a concrete two-layer example that traces the distance computation for one pair of points.
  3. [Sec. 3.6, Theorems 1-2 and unreachable pairs] Theorems 1 and 2 bound the number and size of connected components of the hierarchical graph, but the text's conclusion that these bounds 'imply the rationality of computing geodesic distance' does not follow: a bound on component structure says nothing about whether shortest-path distances on this graph are a good proxy for semantic similarity. More concretely, the implementation sets unreachable pairs to infinity, and if the graph has many connected components, a large fraction of candidate negatives may be excluded from the contrastive loss; the effective number of negatives then differs from the cosine baseline and varies during training. Please report the average fraction of reachable pairs in the graph during pre-training and specify exactly how infinite distances are handled in the angle-normalization and InfoNCE steps (e.g., masking). This is needed to confirm that the comparison with the cosine baseline is not confounded by a change in the number of negatives.
  4. [Sec. 4.5, Table 7] The hyperparameter analysis does not appear to test the pre-training procedure it is meant to justify. The text says that the authors 'use pre-training models without geodesic distance and only fine-tuned using different hyper-parameters for the image-text retrieval,' but ncluster, the number of neighbors, the number of graph layers, and the update period T0 are all hyperparameters of the geodesic module during pre-training. Varying them only at fine-tuning time cannot establish that the proposed pre-training is insensitive to them. Please clarify what was actually varied; if the ablations did not rerun pre-training, they should be rerun for the most consequential parameters (neighbor count and T0), since the headline comparisons in Tables 1-3 use models pre-trained with specific values of these parameters.
minor comments (5)
  1. [Sec. 3.4] The angle-normalization description is ambiguous: truncating the accumulated angle at 4*pi and then normalizing to [0, pi] is not a standard normalization and changes the scale of the distance; please give the exact formula and define how the cosine is applied to the normalized value.
  2. [Sec. 3.6 / Supplementary Sec. 10.1] The notation is inconsistent: Theorem 1 in the main text uses sigma for the number of neighbors, while the proof in the supplement uses kappa, and Eq. (4) has an unbalanced parenthesis; please unify the symbols and typeset the formula correctly.
  3. [Sec. 4.4] The one-epoch fine-tuning protocol for CLIP_FT and FLIP_FT is unconventional and the margins in Table 4 are small; please report the exact protocol, including whether the baselines were fine-tuned for the same number of steps and with the same data order, and report variance across seeds if available.
  4. [Sec. 2.2] The claim that this paper introduces geodesic distance to multimodal learning 'for the first time' should be qualified in light of GraphWalk [75], which proposes a differentiable geodesic distance estimator, and of prior graph-based contrastive methods; a sentence explaining the difference would help.
  5. [Fig. 1] The caption reports cosine and geodesic values that appear to mix similarity and distance scales (0.87 vs 0.24, 0.93 vs 0.91); please clarify whether the numbers are distances or similarities and what the axes of the figure represent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the geodesic-distance claim is tested on external downstream benchmarks and is not derived from its own inputs.

full rationale

The paper's central claim, that replacing cosine distance with a hierarchical-graph geodesic distance in the contrastive loss improves vision-language pre-training, is evaluated on external benchmarks (COCO and Flickr30K retrieval, VQA, NLVR2, SNLI-VE, RefCOCO+) and is not obtained by construction from the loss definition. The geodesic distance is built from cosine trivial distances plus Floyd shortest paths on a hierarchical graph, but it is not defined in terms of the retrieval metrics, and no retrieval labels are used to fit the graph. Hyperparameter choices in Sec. 4.5 are tuned on COCO fine-tuned retrieval, which overlaps with a headline benchmark; this is a disclosed tuning concern that may inflate results, but it is not a by-construction equivalence, since the encoders must still be trained through the pre-training objectives. The paper's self-citations (refs 58-62) are contextual and support general contrastive-learning and symmetry claims; no load-bearing argument reduces to them, and no uniqueness theorem is imported from the authors' prior work. The main manuscript-level weakness, which a skeptical reader identifies, is that Algorithm 2 does not specify a differentiable surrogate, straight-through estimator, or detached-distance weighting for the Floyd shortest-path and nearest-cluster argmin operations, so the gradient path to the encoders is unclear; this is a correctness or reproducibility risk rather than a circularity risk. No step in the derivation defines the claimed output in terms of the inputs, renames a fitted parameter as a prediction, or smuggles the key ansatz in via self-citation. Therefore no significant circularity is found.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central method relies on the queue as a manifold proxy, a locally flat cosine geometry with several tuned constants, and an untested filtering rule for unreachable pairs. No novel physical or mathematical entities are introduced.

free parameters (7)
  • delta in simple manifold assumption (Def. 2) = sqrt(d), d = 256
    Threshold that decides when local neighborhoods are flat enough to use cosine as the trivial metric; chosen in Section 3.5, not derived.
  • number of K-Means cluster centers ncluster = 256
    Selected via retrieval ablation in Table 7(b); 256 gives the best text retrieval R@1 on COCO.
  • number of graph neighbors per cluster center = 8
    Selected via ablation in Table 7(c); values from 6 to 12 all perform similarly.
  • hierarchical graph layers = 2
    Selected via ablation in Table 7(a); deeper layers add time without improving accuracy.
  • graph update period T0 = 100 training steps
    Selected via ablation in Table 7(d); 75 to 125 steps are similar.
  • angle truncation threshold = 4*pi
    Truncates the accumulated geodesic angle before normalization; chosen in Section 3.5 without an ablation.
  • feature queue size = 65536
    Inherited from ALBEF and TCL; it affects graph density and computational cost but is not independently tuned.
assumptions (5)
  • domain assumption The feature queue provides a sample pool dense enough that a kNN graph approximates the underlying data manifold.
    Section 3.1 uses the queue as the reference frame for all distances; if density is insufficient, shortest-path distances become arbitrary.
  • ad hoc to paper Simple Manifold Assumption (Def. 2): within local neighborhoods, the geodesic distance differs from cosine distance by less than delta.
    Definition 2 and Section 3.3 justify replacing local geodesic edges with cosine; this is not verified on data.
  • domain assumption Clustering recursion reaches a bottom layer where each cluster satisfies the simple manifold assumption.
    Section 3.3 assumes bottom-layer clusters are small enough for a trivial metric; if not, Equation 2 degrades.
  • domain assumption The momentum feature queue is stable enough that a graph rebuilt every T0 steps remains valid for distances computed in between.
    Section 3.3 Q3 introduces periodic reconstruction but gives no analysis of drift between reconstructions.
  • ad hoc to paper Unreachable pairs in the graph can be safely treated as very distant negatives.
    Section 3.6 says unreachable pairs are set to infinity to filter overly simple negatives; this changes the loss and is not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GeoMM: On Geodesic Perspective for Multi-modal Learning." pith.science (2026). https://pith.science/paper/EKXTSEJZ

@misc{pith2026250511216,
  author       = {Pith},
  title        = {Pith review of: GeoMM: On Geodesic Perspective for Multi-modal Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKXTSEJZ}},
  note         = {Machine review of arXiv:2505.11216}
}
read the original abstract

Geodesic distance serves as a reliable means of measuring distance in nonlinear spaces, and such nonlinear manifolds are prevalent in the current multimodal learning. In these scenarios, some samples may exhibit high similarity, yet they convey different semantics, making traditional distance metrics inadequate for distinguishing between positive and negative samples. This paper introduces geodesic distance as a novel distance metric in multi-modal learning for the first time, to mine correlations between samples, aiming to address the limitations of common distance metric. Our approach incorporates a comprehensive series of strategies to adapt geodesic distance for the current multimodal learning. Specifically, we construct a graph structure to represent the adjacency relationships among samples by thresholding distances between them and then apply the shortest-path algorithm to obtain geodesic distance within this graph. To facilitate efficient computation, we further propose a hierarchical graph structure through clustering and combined with incremental update strategies for dynamic status updates. Extensive experiments across various downstream tasks validate the effectiveness of our proposed method, demonstrating its capability to capture complex relationships between samples and improve the performance of multimodal learning models.

Figures

Figures reproduced from arXiv: 2505.11216 by the authors.

Figure 1
Figure 1. The motivation of leveraging geodesic metric. Based [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of our proposed hierarchical graph struc [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visual grounding visualization (left) and performance of weakly-supervised visual grounding on RefCOCO+ (right). [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

111 extracted references · 59 canonical work pages

  1. [1]

    Geometry of oblique projections

    E Andruchow, Gustavo Corach, and D Stojanoff. Geometry of oblique projections. arXiv preprint math/9911133, 1999. 3, 1

    work page Pith review arXiv 1999

  2. [2]

    Vqa: Visual question an- swering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question an- swering. In International Conference on Computer Vision, pages 2425–2433, 2015

    2015

  3. [3]

    Geodesic matting: A framework for fast interactive image and video seg- mentation and matting

    Xue Bai and Guillermo Sapiro. Geodesic matting: A framework for fast interactive image and video seg- mentation and matting. Int. J. Comput. Vis. , 82(2): 113–132, 2009. 3

    2009

  4. [4]

    Vlmo: Uni- fied vision-language pre-training with mixture-of- modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Uni- fied vision-language pre-training with mixture-of- modality-experts. Advances in Neural Information Processing Systems, 35:32897–32912, 2022. 1, 2, 6, 7

    2022

  5. [5]

    Grit-vlp: Grouped mini-batch sam- pling for efficient vision and language pre-training

    Jaeseok Byun, Taebaek Hwang, Jianlong Fu, and Taesup Moon. Grit-vlp: Grouped mini-batch sam- pling for efficient vision and language pre-training. In European Conference on Computer Vision, pages 395–412. Springer, 2022

    2022

  6. [6]

    Mafa: Managing false negatives for vision-language pre-training

    Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision-language pre-training. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 27314–27324, 2024. 1, 3, 6, 7

    2024

  7. [7]

    End-to-end object detection with trans- formers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with trans- formers. In European Conference on Computer Vi- sion, pages 213–229. Springer, 2020

    2020

  8. [8]

    Un- supervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912–9924, 2020. 8

    2020

Show all 111 references
  1. [9]

    Emerging properties in self-supervised vi- sion transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vi- sion transformers. In IEEE Conference on Computer Vision and Pattern Recognition , pages 9650–9660, 2021

    2021

  2. [10]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021

    2021

  3. [11]

    Stair: Learning sparse text and image representation in grounded tokens

    Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexan- der Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: Learning sparse text and image representation in grounded tokens. arXiv preprint arXiv:2301.13081, 2023. 2

    2023 arXiv

  4. [12]

    Vlp: A survey on vision-language pre-training

    Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38–56, 2023

    2023

  5. [13]

    A simple framework for con- trastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In In- ternational Conference on Machine Learning, pages 1597–1607. PMLR, 2020. 3

    2020

  6. [14]

    Exploring simple siamese representation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021

    2021

  7. [15]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 8

    2003 arXiv

  8. [16]

    X- volution: On the unification of convolution and self- attention

    Xuanhong Chen, Hang Wang, and Bingbing Ni. X- volution: On the unification of convolution and self- attention. arXiv preprint arXiv:2106.02253, 2021. 2

    2021 arXiv

  9. [17]

    Uniter: Universal image-text represen- tation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text represen- tation learning. In European Conference on Com- puter Vision, pages 104–120. Springer, 2020. 1, 2, 6, 7

    2020

  10. [18]

    Unsupervised opinion summariza- tion using approximate geodesics

    Somnath Basu Roy Chowdhury, Nicholas Monath, Avinava Dubey, Amr Ahmed, and Snigdha Chaturvedi. Unsupervised opinion summariza- tion using approximate geodesics. arXiv preprint arXiv:2209.07496, 2022. 2

    2022 arXiv

  11. [19]

    Geodesics in heat: A new approach to computing distance based on heat flow

    Keenan Crane, Clarisse Weischedel, and Max Wardetzky. Geodesics in heat: A new approach to computing distance based on heat flow. ACM Trans. Graph., 32(5):152:1–152:11, 2013. 2

    2013

  12. [20]

    Imagenet: A large-scale hierar- chical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierar- chical image database. In IEEE Conference on Com- puter Vision and Pattern Recognition , pages 248– 255, 2009. 1

    2009

  13. [21]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2, 5, 1

    2018 arXiv

  14. [22]

    Similarity reasoning and filtration for image-text matching

    Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI conference on artificial intelligence, pages 1218–1226, 2021

    2021

  15. [23]

    A note on two problems in con- nexion with graphs

    Edsger W Dijkstra. A note on two problems in con- nexion with graphs. Numerische mathematik, 1(1): 269–271, 1959

    1959

  16. [24]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  17. [25]

    Algorithm 97: shortest path

    Robert W Floyd. Algorithm 97: shortest path. Com- munications of the ACM, 5(6):345–345, 1962. 2, 4

    1962

  18. [26]

    Large-scale adversar- ial training for vision-and-language representation learning

    Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversar- ial training for vision-and-language representation learning. Advances in Neural Information Process- ing Systems, 33:6616–6628, 2020. 6, 7

    2020

  19. [27]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In IEEE Conference on Com- puter Vision and Pattern Recognition, pages 15180– 15190, 2023. 2

    2023

  20. [28]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020. 1, 3, 5

    2020

  21. [29]

    Masked autoen- coders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll ´ar, and Ross Girshick. Masked autoen- coders are scalable vision learners. In IEEE Confer- ence on Computer Vision and Pattern Recognition , pages 16000–16009, 2022

    2022

  22. [30]

    Geonet: Deep geodesic networks for point cloud analysis

    Tong He, Haibin Huang, Li Yi, Yuqian Zhou, Chihao Wu, Jue Wang, and Stefano Soatto. Geonet: Deep geodesic networks for point cloud analysis. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 6888–6897, 2019. 3

    2019

  23. [32]

    Pixel-bert: Aligning image pixels with text by deep multi-modal transformers

    Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dong- mei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. CoRR, abs/2004.00849, 2020. 2

    2004 arXiv

  24. [33]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Ma- chine Learning, pages 4904–491...

    2021

  25. [34]

    Vilt: Vision-and-language transformer without convolu- tion or region supervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In International Con- ference on Machine Learning , pages 5583–5594. PMLR, 2021. 2, 6, 7, 1

    2021

  26. [35]

    Computing geodesic paths on manifolds

    Ron Kimmel and James A Sethian. Computing geodesic paths on manifolds. Proceedings of the national academy of Sciences , 95(15):8431–8435,

  27. [36]

    Visual genome: Connecting language and vision using crowdsourced dense image annota- tions

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annota- tions. International Journal of Compute...

    2017

  28. [37]

    Numba: a llvm-based python JIT compiler

    Siu Kwan Lam, Antoine Pitrou, and Stanley Seib- ert. Numba: a llvm-based python JIT compiler. In LLVM@SC, pages 7:1–7:6. ACM, 2015. 6

    2015

  29. [38]

    Le, Vu Nguyen, Chen-Ping Yu, and Dimitris Samaras

    Hieu M. Le, Vu Nguyen, Chen-Ping Yu, and Dimitris Samaras. Geodesic distance histogram feature for video segmentation. In Asian Conference on Com- puter Vision, pages 275–290, 2016. 3

    2016

  30. [39]

    Stacked cross attention for image- text matching

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image- text matching. In European Conference on Computer Vision, pages 201–216, 2018

    2018

  31. [40]

    Multimodal foundation models: From spe- cialists to general-purpose assistants

    Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jian- wei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From spe- cialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 2023. 2

    2023 arXiv

  32. [41]

    Align before fuse: Vision and lan- guage representation learning with momentum dis- tillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Got- mare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and lan- guage representation learning with momentum dis- tillation. Advances in Neural Information Processing Systems, 34:9694–9705, 2021. 1, 2...

    2021

  33. [42]

    Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Ma- chine Learning, pages 12888–12900. PMLR, 2022. 2

    2022

  34. [43]

    Hero: Hierarchical en- coder for video+ language omni-representation pre- training

    Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical en- coder for video+ language omni-representation pre- training. arXiv preprint arXiv:2005.00200, 2020

    2005 arXiv

  35. [44]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 1, 2

    1908 arXiv

  36. [45]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. InEu- ropean Conference on Computer Vision, pages 121– 137, 2020. 1, 2, 6, 7

    2020

  37. [46]

    Supervision exists everywhere: A data efficient contrastive language-image pre- training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre- training paradigm. arXiv preprint arXiv:2110.05208, 2021

    arXiv

  38. [47]

    Scaling language- image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre-training via masking. In IEEE Conference on Computer Vision and Pattern Recognition, pages 23390–23400, 2023. 2, 8

    2023

  39. [48]

    Geodesic self- attention for 3d point clouds

    Zhengyu Li, Xuan Tang, Zihao Xu, Xihao Wang, Hui Yu, Mingsong Chen, and Xian Wei. Geodesic self- attention for 3d point clouds. In Advances in Neural Information Processing Systems, 2022. 3

    2022

  40. [49]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Confer- ence on Computer Vision, pages 740–755. Springer,

  41. [50]

    Haibin Ling and David W. Jacobs. Deformation invariant image matching. In IEEE International Conference on Computer Vision , pages 1466–1473,

  42. [51]

    Adap- tive reconstruction network for weakly supervised re- ferring expression grounding

    Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Dechao Meng, and Qingming Huang. Adap- tive reconstruction network for weakly supervised re- ferring expression grounding. In IEEE International Conference on Computer Vision , pages 2611–2620. IEEE, 2019. 7, 8

    2019

  43. [52]

    Algorithm as 136: A k-means clustering algorithm

    Ilya Loshchilov and Frank Hutter. Algorithm as 136: A k-means clustering algorithm. pages 100–108,

  44. [53]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations. OpenReview.net, 2019. 5

    2019

  45. [54]

    Vilbert: Pretraining task-agnostic visiolinguistic rep- resentations for vision-and-language tasks.Advances in Neural Information Processing Systems, 32, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic rep- resentations for vision-and-language tasks.Advances in Neural Information Processing Systems, 32, 2019. 1, 2

    2019

  46. [55]

    Computing geodesics on triangular meshes.Comput- ers & Graphics, 29(5):667–675, 2005

    Dimas Mart ´ınez, Luiz Velho, and Paulo C Carvalho. Computing geodesics on triangular meshes.Comput- ers & Graphics, 29(5):667–675, 2005

    2005

  47. [56]

    Bron- stein, and Pierre Vandergheynst

    Jonathan Masci, Davide Boscaini, Michael M. Bron- stein, and Pierre Vandergheynst. Geodesic convolu- tional neural networks on riemannian manifolds. In IEEE International Conference on Computer Vision Workshops, pages 832–840, 2015. 3

    2015

  48. [57]

    Jensen’s inequality

    Edward James McShane. Jensen’s inequality. 1937. 2

    1937

  49. [58]

    Towards bridging sample complexity and model capacity

    Shibin Mei, Chenglong Zhao, Shengchao Yuan, and Bingbing Ni. Towards bridging sample complexity and model capacity. InProceedings of the AAAI Con- ference on Artificial Intelligence , pages 1972–1980,

    1972

  50. [59]

    Towards interpreting and utiliz- ing symmetry property in adversarial examples

    Shibin Mei, Chenglong Zhao, Bingbing Ni, and Shengchao Yuan. Towards interpreting and utiliz- ing symmetry property in adversarial examples. In Proceedings of the AAAI Conference on Artificial In- telligence, pages 9126–9133, 2023. 3

    2023

  51. [60]

    Exploring and utilizing pattern imbal- ance

    Shibin Mei, Chenglong Zhao, Shengchao Yuan, and Bingbing Ni. Exploring and utilizing pattern imbal- ance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7569–7578, 2023

    2023

  52. [61]

    Mssidd: A benchmark for multi-sensor denoising

    Shibin Mei, Hang Wang, and Bingbing Ni. Mssidd: A benchmark for multi-sensor denoising. arXiv preprint arXiv:2411.11562, 2024. 3

    2024 arXiv

  53. [62]

    Object- oriented anchoring and modal alignment in multi- modal learning

    Shibin Mei, Bingbing Ni, Hang Wang, Chenglong Zhao, Fengfa Hu, Zhiming Pi, and Bilian Ke. Object- oriented anchoring and modal alignment in multi- modal learning. In European Conference on Com- puter Vision, pages 179–196. Springer, 2025. 2

    2025

  54. [63]

    Joseph S. B. Mitchell, David M. Mount, and Chris- tos H. Papadimitriou. The discrete geodesic problem. SIAM J. Comput., 16(4):647–668, 1987. 2

    1987

  55. [64]

    Analytic inequalities

    Dragoslav S Mitrinovic and Petar M Vasic. Analytic inequalities. Springer, 1970. 2

    1970

  56. [65]

    Slip: Self-supervision meets language- image pre-training

    Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In European Conference on Computer Vision, pages 529–544. Springer, 2022

    2022

  57. [66]

    Geodesic-former: A geodesic-guided few-shot 3d point cloud instance segmenter

    Tuan Ngo and Khoi Nguyen. Geodesic-former: A geodesic-guided few-shot 3d point cloud instance segmenter. In European Conference on Computer Vi- sion, pages 561–578, 2022. 3

    2022

  58. [67]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  59. [68]

    Im2text: Describing images using 1 million cap- tioned photographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million cap- tioned photographs. Advances in Neural Information Processing Systems, 24, 2011. 6

    2011

  60. [69]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gre- gory Chanan, Edward Yang, Zachary DeVito, Zem- ing Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Ad- vances in Neural Information Processing Systems Workshop, 2017

    2017

  61. [70]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366, 2022

    2022 arXiv

  62. [71]

    Computational optimal transport: With applications to data science

    Gabriel Peyr ´e, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11 (5-6):355–607, 2019

    2019

  63. [72]

    Combined scaling for zero-shot transfer learn- ing

    Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learn- ing. Neurocomputing, 555:126658, 2023. 2

    2023

  64. [73]

    Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models. In International Conference on Computer Vision, pages 2641–2649, 2015. 7

    2015

  65. [74]

    Straightest geodesics on polyhedral surfaces

    Konrad Polthier and Markus Schmies. Straightest geodesics on polyhedral surfaces. In ACM SIG- GRAPH 2006 Courses, pages 30–38. 2006

    2006

  66. [75]

    Graphwalks: Efficient shape agnostic geodesic shortest path estimation

    Rolandos Alexandros Potamias, Alexandros Ne- ofytou, Kyriaki-Margarita Bintsi, and Stefanos Zafeiriou. Graphwalks: Efficient shape agnostic geodesic shortest path estimation. In IEEE Confer- ence on Computer Vision and Pattern Recognition Workshop, pages 2967–2976, 2022. 3

    2022

  67. [76]

    Imagebert: Cross-modal pre- training with large-scale weak-supervised image-text data

    Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre- training with large-scale weak-supervised image-text data. CoRR, abs/2001.07966, 2020. 6

    2001 arXiv

  68. [77]

    Improving language under- standing by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training

  69. [78]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pa...

    2021

  70. [79]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015

    2015

  71. [80]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Ab- hishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In International Conference on Computer Vision , pages 618–626, 2017. 7

    2017

  72. [81]

    Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics, pages 2556–2565, 2018. 6

    2018

  73. [82]

    Vl-bert: Pre-training of generic visual-linguistic representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530, 2019. 2

    1908 arXiv

  74. [83]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023. 2

    2023 arXiv

  75. [84]

    A corpus for reason- ing about natural language grounded in photographs

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reason- ing about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018. 7

    2018 arXiv

  76. [85]

    Revisiting unreasonable effective- ness of data in deep learning era

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effective- ness of data in deep learning era. In International Conference on Computer Vision , pages 843–852, 2017

    2017

  77. [86]

    Gortler, and Hugues Hoppe

    Vitaly Surazhsky, Tatiana Surazhsky, Danil Kir- sanov, Steven J. Gortler, and Hugues Hoppe. Fast exact and approximate geodesics on meshes. ACM Trans. Graph., 24(3):553–560, 2005. 2

    2005

  78. [87]

    LXMERT: learning cross-modality encoder representations from trans- formers

    Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from trans- formers. In EMNLP/IJCNLP, pages 5099–5110. As- sociation for Computational Linguistics, 2019. 2

    2019

  79. [88]

    Tenenbaum, Vin de Silva, and John C

    Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. A global geometric framework for non- linear dimensionality reduction. Science, 290(5500): 2319–2323, 2000. 2, 3

    2000

  80. [89]

    Pigeon hole principle

    Wojciech A Trybulec. Pigeon hole principle. Journal of Formalized Mathematics, 2(199):0, 1990. 2

    1990

  81. [90]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. 1, 2

    2017

  82. [91]

    Optimal transport: old and new

    C ´edric Villani et al. Optimal transport: old and new. Springer, 2009

    2009

  83. [92]

    Learning to combine: Knowledge aggrega- tion for multi-source domain adaptation

    Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggrega- tion for multi-source domain adaptation. In Euro- pean Conference on Computer Vision, 2020. 3

    2020

  84. [93]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subho- jit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022. 1, 2

    2022 arXiv

  85. [94]

    Mvp: Multimodality-guided visual pre-training

    Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality-guided visual pre-training. In European Conference on Computer Vision, pages 337–353, 2022

    2022

  86. [95]

    Visual entailment: A novel task for fine-grained image understanding

    Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706, 2019. 7

    1901 arXiv

  87. [96]

    A fast proximal point method for computing exact wasserstein distance

    Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. In Un- certainty in Artificial Intelligence , pages 433–453. PMLR, 2020

    2020

  88. [97]

    Vision-language pre- training with triple contrastive learning

    Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sam- path Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre- training with triple contrastive learning. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 15671–15680, 2022. 1...

    2022

  89. [98]

    Unified contrastive learning in image-text-label space

    Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In IEEE Conference on Computer Vision and Pattern Recognition, pages 19163–19173, 2022

    2022

  90. [99]

    Filip: Fine-grained in- teractive language-image pre-training.arXiv preprint arXiv:2111.07783, 2021

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained in- teractive language-image pre-training.arXiv preprint arXiv:2111.07783, 2021. 6

    2021 arXiv

  91. [100]

    FILIP: fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine-grained interactive language-image pre-training. In ICLR. OpenReview.net, 2022

    2022

  92. [101]

    Coca: Contrastive captioners are image-text founda- tion models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text founda- tion models. arXiv preprint arXiv:2205.01917, 2022. 1, 2

    2022 arXiv

  93. [102]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexan- der C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69–85. Springer, 2016. 7

    2016

  94. [103]

    Flo- rence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Flo- rence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021

    2021 arXiv

  95. [104]

    Lit: Zero-shot transfer with locked- image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked- image text tuning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022

    2022

  96. [105]

    Vinvl: Revisiting visual representa- tions in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representa- tions in vision-language models. InIEEE Conference on Computer Vision and Pattern Recognition, pages 5579–5588, 2021. 1, 2

    2021

  97. [106]

    Learning 2d temporal adjacent networks for moment localization with natural language

    Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI Conference on Artificial In- telligence, pages 12870–12877, 2020

    2020

  98. [107]

    Counterfactual contrastive learning for weakly- supervised vision-language grounding

    Zhu Zhang, Zhou Zhao, Zhijie Lin, Xiuqiang He, et al. Counterfactual contrastive learning for weakly- supervised vision-language grounding. Advances in Neural Information Processing Systems , 33:18123– 18134, 2020. 8 GeoMM: On Geodesic Perspective for Multi-modal Learning Supp...

    2020

  99. [108]

    Tak- ing [41] as an example, the network includes an image en- coder, a text encoder, and a multimodal fusion encoder

    Network Structure Our model structure mainly follows [41], [97] and [6]. Tak- ing [41] as an example, the network includes an image en- coder, a text encoder, and a multimodal fusion encoder. The image encoder is a 12-layer transformer with a VIT struc- ture, with initialized ...

  100. [109]

    COCO VG SBU CC3M #image 113K 100K 860K 2.95M #text 567K 769K 860K 2.95M Table 8

    Pre-training Datasets The details of the pre-training datasets about image-text pairs are shown in below Tab.8. COCO VG SBU CC3M #image 113K 100K 860K 2.95M #text 567K 769K 860K 2.95M Table 8. Statistics of the pre-training datasets

  101. [110]

    Comparison with Oblique manifold

    Pre-training Tasks Multimodal learning requires elaborate pre-training tasks, and commonly used pre-training tasks include Masked Language Modeling(MLM) [21], Image-Text Matching (ITM) [41], Image-Text Contrastive (ITC) [78], Word Patch Method TR IR ALBEF 73.1 56.8 ALBEF+OM 73...

  102. [111]

    We conduct the experiments on the image-text retrieval task with the COCO dataset and fine-tune setting

    Extra Experiments We also compare our method with the Oblique manifold (OM) [1]. We conduct the experiments on the image-text retrieval task with the COCO dataset and fine-tune setting. We display the R@1 accuracy for text retrieval (TR) and image retrieval (IR), as shown in Tab.9

  103. [112]

    Proofs 10.1. Proof for Theorem 1 For the graph where the cluster centers represent the ver- tices of the graph and the adjacent relationship between these cluster centers represents the edges between vertices, we can know this graph possessesN vertices with minimum degreeκ (cl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.