Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Model alignment using inter-modal bridges

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The paper claims that a flow trained on fewer than 20% paired samples can fuse pre-trained models across modalities — vision, text, and brain recordings — matching end-to-end accuracy on recognition and generation.

desk verdict Useful empirical study of alignment strategies, but the bridge-cost advantage rests on a circular evaluation. read the letter →

arxiv 2505.12322 v1 pith:BQ6SYUPF submitted 2025-05-18 cs.LG cs.CV

classification cs.LGcs.CV
keywords modelalignmentconditionalflowmatchinginter-modalbridgecostoptimaltransportsemi-supervisedlearninglatentspacemultimodalrepresentationneural
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that two pre-trained models living in different modalities — a vision encoder and a text encoder, or a deep network and a patch of monkey visual cortex — can be fused by a single learned flow between their latent spaces, and that this flow needs fewer than 20% of samples to be paired in order to match the downstream accuracy of models trained end-to-end on both modalities. The alignment is computed in one of two ways: directly from the labelled pairs, or by solving an optimal transport problem whose cost matrix is built from a new bridge cost that treats the sparse paired samples as zero-cost anchors and expresses every cross-space distance as a chain of within-space distances. The paper reports parity with end-to-end baselines on MNIST image-to-text and text-to-image tasks, on ImageNet image-to-text classification measured against the vision transformer's own classification head, and on category decoding from mapped neural activity. The reason this would matter is modularity: if the claim holds, frozen feature extractors can be coupled cheaply, without retraining either model, whenever a small anchor set is available.

What carries the argument

The load-bearing object is the inter-modal bridge cost, a rule that converts sparse labels into a dense cross-space distance matrix: $C^{XY}_{\text{bridge}}(x_i, y_j) = 0$ when the pair is a labelled anchor, and otherwise $C^{XY}_{\text{bridge}}(x_i, y_j) = \min_{(x^p_i, y^p_j) \in P} \big[ C_{XX}(x_i, x^p_i) + C_{YY}(y^p_j, y_j) \big]$, with within-space costs taken as cosine distances (or correlation-based costs for neural data). The work this cost does is to make two incomparable latent spaces comparable so an entropic optimal transport solver — linear, fused Gromov–Wasserstein, or unbalanced — can produce a coupling $\pi^*$; conditional flow matching then trains the velocity field $v_{t,\theta}$ against the conditional vector field $u_t(x \mid x_1)$, and out-of-sample predictions come from solving the induced ODE from noise to target conditioned on the source point. The velocity field is parameterised with adaptive-normalisation blocks conditioned on time and source input, which the paper shows learn faster and reach higher accuracy than the plain multi-layer-perceptron architecture of the cross-domain flow-matching method it builds on.

What would settle it

Take a source encoder whose latent space is deliberately scrambled (for example, a vision transformer with class tokens randomly permuted) so within-space distances no longer track class identity, keep the same paired anchors, and run the full pipeline; if coupling accuracy and downstream accuracy do not collapse, the intra-space geometry is not what carries the alignment. A complementary check on the same principle: on a dataset with full labels, compute the bridge-cost coupling from a random 10% anchor set and count how many of the highest-weight coupling entries match ground-truth correspondences, comparing against the fused costs the paper benchmarks in its Figure 6.

Watch

Extended reading notes

Core claim

The paper's central claim is that inter-modal alignment can be decoupled from representation learning: once each side has a frozen pre-trained encoder, the correspondence between their latent spaces is itself a learnable object. Its vehicle is a new inter-modal bridge cost $C^{XY}_{\text{bridge}}(x_i, y_j)$ over pairs from two spaces that share no common metric: paired samples $(x^p_i, y^p_j)$ get cost zero and act as anchors, and any other pair is charged the minimum over anchors of the within-space distance from $x_i$ to its anchor plus the within-space distance from the anchor's partner to $y_j$. This dense cost matrix feeds a balanced or unbalanced entropic optimal transport solver whose coupling $\pi^*$ defines the training pairs for a conditional flow: a time-dependent velocity field $v_{t,\theta}(y \mid x)$ is trained to morph noise into the target distribution conditioned on a source point, giving out-of-sample transport by solving an ODE. With under 20% of samples paired, the paper reports downstream accuracy matching end-to-end trained models on object recognition across MNIST and ImageNet and on MNIST image generation, and it reports that on noisy biological data the global transport alignment improves as pairing grows while direct paired alignment overfits.

Load-bearing premise

The method rests on the pre-trained feature extractors doing their job: if the source latent space does not separate classes by distance — the paper measures this as feature-space overlap and shows performance declining as overlap grows — the bridge cost and the learned flow carry no reliable signal, so alignment quality is bounded by the upstream models' capacity.

Editorial extensions

If this is right

  • With fewer than 20% of samples paired, the learned flow matches end-to-end trained baselines on MNIST image-to-text and text-to-image tasks, and outperforms them below about 10% pairing.
  • On ImageNet, image-to-text accuracy rises with the paired fraction and reaches a level comparable to the vision transformer's own classification head at roughly 10% pairing, while per-batch local alignment fails to converge within the 18-hour training budget.
  • On biological-to-artificial alignment, global transport alignment's test error falls as pairing grows, whereas direct paired alignment overfits with more pairs — favouring the optimal-transport route on noisy neural data.
  • The bridge cost yields higher coupling matching accuracy than fused costs built from k-nearest-neighbour graphs or kernel canonical correlation analysis, with the best discrete solver (fused Gromov–Wasserstein on MNIST, linear on ImageNet) depending on the dataset.
  • Category decoding from activations mapped to IT and V4 tracks Brain-Score rankings, and the layer-to-region pattern — later layers fit IT, early-to-middle layers fit V4 — matches the ventral-stream hierarchy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper leaves implicit: connect two encoders never seen together (for example, a newly released vision encoder to an existing text encoder) using a few hundred anchors and check whether the same hyperparameters transfer; if the method is truly modular, they should.
  • As the number of anchors grows, the bridge cost should approach the geodesic-style fused cost built from k-nearest-neighbour graphs in the appendix; verifying that downstream accuracy saturates smoothly with anchor count, rather than requiring a minimum threshold, would pin down the cost's scaling behaviour.
  • The under-20% claim assumes anchors are sampled at random, while real deployments often pair only easy classes; a testable extension is to measure accuracy when anchors cover only a subset of classes, since unseen classes would then be routed entirely through within-space distances — exactly the regime the min-over-anchors construction is designed for.
  • The measured link between source-space overlap and flow quality suggests a selection rule the paper does not state: when several pre-trained encoders are available for a modality, use the one with the most separated latent classes as the source; the six-network neural experiment already contains the data to rank encoders and check this against test error.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a semi-supervised method for aligning latent spaces of pre-trained models across modalities, using conditional flow matching. Paired samples are used in two ways: as direct anchors for 'true' alignment, or as zero-cost bridges in a novel inter-modal cost matrix that is fed into an optimal transport solver to obtain a 'global' coupling. The learned flow is then used to morph between latent spaces, and the method is evaluated on image-text alignment (MNIST, ImageNet) and on aligning artificial neural network representations to neural recordings (Majaj et al., 2015). The central claim is that, with less than 20% paired data, the method matches the downstream task performance of end-to-end trained models on object recognition and image generation.

Significance. If the central claim holds, the paper offers a practical, lightweight route to reusing pre-trained encoders across modalities with very little paired data, and its biological-to-artificial alignment results would be a useful contribution to the neural-predictivity literature. The paper's strengths are its broad benchmark coverage, five-seed experiments, and explicit ablations of cost functions, OT solvers, and velocity-field architectures. The feature-overlap analysis in Sec. 5.1.2 and Fig. 5 also makes a falsifiable prediction connecting representation quality to morph quality. However, the global-alignment branch currently rests on a partly circular evaluation and on unspecified large-scale OT details, so the headline claim is stronger than the evidence.

major comments (4)
  1. [§5.1.3, Fig. 6; App. L] The evaluation of the bridge cost appears circular. Eq. (4) assigns zero cost to every pair in the anchor set P, and Fig. 6 reports matching accuracy by sampling from the optimal coupling and checking label agreement. Nothing in the caption or text indicates that the anchor pairs used to build the cost are excluded or held out when computing this accuracy. Because anchor pairs are label-consistent by construction, an OT solver will concentrate mass on them, so a high matching score may reflect the cost construction rather than generalization to unseen pairs. Please re-run the comparison in Fig. 6 with a held-out paired set (or by scoring only non-anchor pairs), and report both anchor and non-anchor accuracy. If the bridge advantage over KNN/KCCA disappears, the global-alignment branch of the abstract's 'under both settings' claim no longer has independent support.
  2. [§5.1.5, Fig. 8] The central claim that the method 'matches downstream task performance ... particularly when labelled training data is scarce (<20%)' is only partially supported by the ImageNet results. The text states that local alignment failed to converge and that the global solver underperformed with very few paired samples, improving to a level comparable to the classifier only at roughly 10% paired data. The figure and text also do not report the true-alignment curve for ImageNet separately, even though 'both settings' is claimed. Please report the true-alignment ImageNet results explicitly, or revise the claim to state which settings and data fractions are actually supported.
  3. [§3.3, Eq. (4)] Eq. (4) is not well-defined as written. The anchor set P contains pairs of the form (x^p_i, y^p_j), but the second line of the minimum is written over '(x^p_i, x^p_j) ∈ P' and then uses C_YY(y^p_j, y_j), mixing source and target indices. This should be rewritten as a minimum over anchor indices k of C_XX(x_i, x^p_k) + C_YY(y^p_k, y_j). Since this equation defines the proposed bridge cost, the notation needs to be corrected before the method can be implemented or reproduced.
  4. [§5.1.5, App. E] The scalability of the global-alignment experiments is not specified. The text says global alignment computes an OT plan over the full dataset, but the ImageNet training split is 50% of roughly 1.2M images (about 600K points), for which a dense pairwise cost matrix is infeasible. Please state the exact number of points used for the global OT solve in Fig. 8, the subsampling procedure, and the discrete solver (e.g., Sinkhorn on a subsample, or a low-rank method). This information is necessary to reproduce the reported results and to assess the scalability claims.
minor comments (5)
  1. [Section 1] The contributions list contains two items labelled '(b)' and no '(c)'; please renumber the list.
  2. [Algorithm 1, App. E] The pseudocode computes πtrue for true alignment and then unconditionally overwrites π with an OT coupling, so it does not reflect the described three strategies; add explicit conditional branches for true, global, and local alignment.
  3. [§5.1.3] The sentence 'we assessed ... (Sec 5.1.3) (Sec 5.1.4)' contains a duplicated parenthetical reference; please clean up the cross-references.
  4. [Figs. 6, 8, 9] The 'Paired:Unpaired' axis labels are not defined in the main text or captions; please state whether the values are fractions of the full training set and define the exact ratios used.
  5. [Throughout] The initials 'U-GENOT' and 'GENOT' are used interchangeably; please use one consistent name for the base method.

Circularity Check

1 steps flagged · score 6.0 of 10

Fig. 6 bridge-cost comparison is partly self-confirming: Cbridge assigns zero cost to the same paired anchors on which matching accuracy is scored.

  1. self definitional [Section 3.3, Eq. (4); Section 5.1.3, Fig. 6 caption]
    "Cbridge XY (xi,yj) = 0 if (xp_i,yp_j)∈P; otherwise min_{(xp_i,xp_j)∈P} {CXX(xi,xp_i)+CYY(yp_j,yj)} ... Matching accuracy was calculated by sampling from the optimal coupling π⋆ and averaging the number of correct matches."

    Eq. (4) defines Cbridge to have zero cost on every anchor pair in P, and in Sec. 5.1.3 the optimal coupling π* is the OT solution for that cost. The Fig. 6 caption says matching accuracy is computed by sampling from π*. Since the anchor pairs are label-consistent by construction (paired samples of the same object), any mass placed on these zero-cost entries is automatically counted as a correct match. The paper does not report excluding P or scoring on a held-out paired set, so the reported bridge advantage over KNN/KCCA partly re-states the definition of Cbridge rather than measuring generalization. This comparison is then used to select the bridge cost for all subsequent global-alignment experiments, making that selection partially circular.

full rationale

The paper's downstream claims are not wholly circular: true alignment and global alignment are evaluated on held-out test splits (e.g., MNIST test data, ImageNet 40% evaluation split), so those results are independent of the bridge-cost construction. However, the justification for adopting the bridge cost for global alignment rests on Sec. 5.1.3 / Fig. 6, where the cost is built from the paired set P and matching accuracy is scored on the optimal coupling derived from that same cost. Because anchor pairs have zero cost and are label-correct by definition, the bridge cost's apparent superiority over KNN and KCCA is partly an artifact of the evaluation. No self-citation chain or uniqueness-import argument is present, and the true-alignment branch retains independent empirical content. The partial circularity is confined to the coupling-matching comparison that motivates the bridge cost, so a score of 6 is appropriate.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method relies on a handful of hyperparameters (alpha, tau, epsilon) that are tuned per dataset, and a set of modeling assumptions about paired samples and pre-trained representations. The paper does not introduce new physical or mathematical entities.

free parameters (3)
  • alpha (fusion parameter for FGW) = 0.25 to 0.5 depending on cost and dataset (Appendix L)
    The trade-off parameter between quadratic and linear OT terms in Eq. 9 is selected based on matching accuracy on the same data used to evaluate the coupling.
  • tau_X and tau_Y (unbalanced OT weights) = 0.99 for all neural experiments (Appendix M.2)
    Chosen by fitting an OLS regression on the validation set to maximize the aes ratio, so the value is tuned to the data.
  • epsilon (entropy regularization) = 5e-3 for image-text, 1e-3 for neural (Appendix E, M.2)
    Chosen by hand for each setting, affecting the smoothness of the OT coupling.
assumptions (3)
  • domain assumption The pre-trained feature extractors provide latent spaces where intra-space cosine distances are meaningful for cross-modal matching.
    The bridge cost (Eq. 4) and all OT solvers rely on intra-space distances (C_XX, C_YY) to compute the inter-space cost. If these distances do not reflect semantic similarity, the coupling will be poor.
  • domain assumption A small set of paired samples P serves as a valid anchor linking the two latent spaces.
    The bridge cost assumes paired points have zero cost and that all other points can be related through these anchors. This is only valid if the pairs are representative and correctly matched.
  • standard math The conditional flow matching objective in Eq. 2 can learn a valid transport map from noise to the target space conditioned on source points.
    This is the standard U-GENOT objective from Klein et al. [2023]; the paper relies on the theoretical guarantees of OT-CFM.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model alignment using inter-modal bridges." pith.science (2026). https://pith.science/paper/BQ6SYUPF

@misc{pith2026250512322,
  author       = {Pith},
  title        = {Pith review of: Model alignment using inter-modal bridges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQ6SYUPF}},
  note         = {Machine review of arXiv:2505.12322}
}
abstract

Foundation models have demonstrated remarkable performance across modalities such as language and vision. However, model reuse across distinct modalities (e.g., text and vision) remains limited due to the difficulty of aligning internal representations. Existing methods require extensive paired training data or are constrained to specific domains. We introduce a semi-supervised approach for model alignment via conditional flow matching. The conditional flow between latent spaces of different modalities (e.g., text-to-image or biological-to-artificial neuronal activity) can be learned in two settings: ($1$) solving a (balanced or unbalanced) optimal transport problem with an inter-space bridge cost, and ($2$) performing memory-efficient alignment using labelled exemplars. Despite being constrained by the original models' capacity, our method--under both settings--matches downstream task performance of end-to-end trained models on object recognition and image generation tasks across MNIST, ImageNet, and \cite{majaj2015simple} datasets, particularly when labelled training data is scarce ($<20\%$). Our method provides a data-efficient solution for inter-modal model alignment with minimal supervision.

Figures

Figures reproduced from arXiv: 2505.12322 by the authors.

Figure 1
Figure 1. Pictorial representation of our approach for aligning model space using inter-modal bridges. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pictorial representation of alignment methods. Here, — represents true pairs, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pictorial representation of bridge cost via [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Noise distribution trajectory to the target latent space of a language model at [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Latent space overlaps on conditional flow matching from image-to-text domain, using [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Matching accuracy across the inter-space costs for linear and FGW solvers. The optimised [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Velocity Field vθ architecture benchmark. Comparison of the baseline feed-forward architecture and the adaptive normalisation architecture (with blocks) for image-to-text tasks in a fully supervised setting, using true alignment, across various sizes. Here, A) MNIST ex…
Figure 8
Figure 8. Figure 8: Image-to-text test accuracy for Ima￾geNet. For the ImageNet dataset, we evaluated the accuracy of the image-to-text feature space using varying num￾bers of paired samples ( [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Downstream task performance across different alignment strategies using bridge cost with [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Test mean squared error of the learned conditional flow matching from representations of [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Downstream task performance for image category classification using the learned condi [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Iterations required to achieve 80% accuracy in MNIST experiments. • Reweighting neural networks ηθ, ξθ: multi-layer perceptron (MLP) used in Klein et al. [2023]. Out-of-sample prediction At inference time, we solved Eq. 10 for t1 = 1 using the velocity field vt,θ, a s…
Figure 13
Figure 13. Figure 13: Overview of KNN fused cost. We construct a fused graph from K-Nearest Neighbour [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Overview of KCCA fused cost. Using paired samples, we find the projection vectors [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Mapping from Swiss roll to Spiral using local alignment [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Runtime per iteration (in seconds) for different alignment strategies across MNIST and [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Matching accuracy across different values of [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Matching accuracy across different values of [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Predicted versus true aes ratio. N Additional results for biological-artificial neural representation alignment Here, we provide a breakdown of the results presented in Sec. 5.2. Specifically, the test mean squared error of the learned conditional flow mappings ( [PI…
Figure 20
Figure 20. Figure 20: Heatmaps of the average values of the aes ratio as a function of [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]
Figure 21
Figure 21. Figure 21: Test mean squared error of the learned conditional flow mappings from representations of [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: Downstream image category classification performance using learned conditional flow [PITH_FULL_IMAGE:figures/full_fig_p031_22.png]
Figure 23
Figure 23. Figure 23: Boxplots of test mean squared error of the learned conditional flow matching from latent [PITH_FULL_IMAGE:figures/full_fig_p032_23.png]
Figure 24
Figure 24. Figure 24: Boxplots of test accuracy of the learned conditional flow matching from latent representa [PITH_FULL_IMAGE:figures/full_fig_p033_24.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 22 canonical work pages

  1. [1]

    Are we done with imagenet? arXiv preprint arXiv:2006.07159,

    Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159,

  2. [2]

    For each, we had three variations–small, medium, and large (Table 1)– with the SiLU activation function applied after every layer in all models

    adaLN: blocks with adaptive layer normalisation (adaLN) following [Perez et al., 2018]. For each, we had three variations–small, medium, and large (Table 1)– with the SiLU activation function applied after every layer in all models. For each setting, we measured the number of iterations required to achieve 80% accuracy in image-to-text experiments on the ...

  3. [4]

    Here, FC is for the fully connected layer, Conv is for the convolutional layer and ConvT is for the convolutional transpose layer

    Table 3: Baseline models architectures for the MNIST experiments. Here, FC is for the fully connected layer, Conv is for the convolutional layer and ConvT is for the convolutional transpose layer. all pre-trained on the ImageNet dataset, and the weights were obtained from the PyTorch li- brary [Paszke et al., 1912]. These models were selected for their hi...

  4. [6]

    Forp,q∈P (X), KL(p∥ q) = R Xp(x) log p(x) q(x)dx denotes the Kullback–Leibler divergence between these two distributions

    The entropy for π∈P (X×Y ) is given byH(π) =− R X×Y π(x,y ) log(π(x,y ))d(x,y ). Forp,q∈P (X), KL(p∥ q) = R Xp(x) log p(x) q(x)dx denotes the Kullback–Leibler divergence between these two distributions. For the discrete setting, the Sinkhorn algorithm [Cuturi, 2013] solves the linear entropic OT problem by iteratively updating the coupling to minimise the...

  5. [7]

    Gemini: a family of highly capable multimodal models

    Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

  6. [10]

    Representation Alignment in Neural Networks

    Ehsan Imani, Wei Hu, and Martha White. Representation alignment in neural networks. arXiv preprint arXiv:2112.07806,

  7. [11]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings,

  8. [13]

    GENOT: Entropic (Gromov) Wasserstein Flow Matching with Applications to Single-Cell Genomics

    11 Dominik Klein, Théo Uscidda, Fabian Theis, and Marco Cuturi. Generative entropic neural optimal transport to map within and across spaces. arXiv preprint arXiv:2310.09254,

Show all 41 references
  1. [14]

    Large scale learning of general visual representations for transfer

    Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXiv:1912.11370, 2(8),

  2. [16]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426,

  3. [18]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748,

  4. [19]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

  5. [20]

    arXiv preprint arXiv:1912.01703, 10,

  6. [21]

    for all the extracted layers of the selected models (Table 4), and the corresponding downstream image category classification performance (Fig. 22). Fig. 23 and Fig. 24 present an aggregate of these results. 28 0.1 0.5 0.9 0.99 0.999 0.9999 1.0 τ  1.01 1.01 1.01 0.80 0.33 0.3...

  7. [22]

    URL https://arxiv.org/abs/1908. 10084. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Intern...

  8. [23]

    Meyer Scetbon, Gabriel Peyré, and Marco Cuturi

    doi: 10.1007/s11263-015-0816-y. Meyer Scetbon, Gabriel Peyré, and Marco Cuturi. Linear-time gromov wasserstein distances using low rank couplings and costs. In International Conference on Machine Learning, pages 19347– 19365. PMLR,

  9. [24]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,

  10. [26]

    However, it may also pose risks, such as enabling surveillance via cross- modal linking of personal data or amplifying biases when aligning poorly disentangled representations

    14 Supplementary Materials: Model alignment using inter-modal bridges A Societal impact Our method improves model reuse across modalities with minimal supervision, enabling applications in low-resource settings. However, it may also pose risks, such as enabling surveillance vi...

  11. [28]

    Additionally, regularisation can improve convergence properties and ensure the existence of unique solutions[Peyré and Cuturi, 2019]

    improves computational efficiency and stability by making the problem more tractable, often leading to smoother and more robust solutions. Additionally, regularisation can improve convergence properties and ensure the existence of unique solutions[Peyré and Cuturi, 2019]. (Unb...

  12. [29]

    This approach necessitates only a small amount of labelled data but requires an additional model to be trained in the joint space for downstream tasks

    proposed to learn a shared embedding space by mapping the representations from two pre-trained multi-modal models into a common space. This approach necessitates only a small amount of labelled data but requires an additional model to be trained in the joint space for downstre...

  13. [30]

    • Reweighting neural networksηθ,ξθ: multi-layer perceptron (MLP) used in Klein et al

    17 Figure 12: Iterations required to achieve 80% accuracy in MNIST experiments. • Reweighting neural networksηθ,ξθ: multi-layer perceptron (MLP) used in Klein et al. [2023]. Out-of-sample prediction At inference time, we solved Eq. 10 fort1 = 1 using the velocity field vt,θ, a...

  14. [31]

    MLP (i.e., Baseline): three separate blocks for latent noise, time, and condition, which were concatenated and processed by a final MLP block following [Klein et al., 2023], and

  15. [33]

    baseline. F.1 Adaptive layer normalisation Diffusion Transformers (DiT) [Peebles and Xie, 2023] and earlier works on diffusion models with U-net backbones [Dhariwal and Nichol, 2021] demonstrated the effectiveness of adaLN. For our formulation, we similarly replaced the standa...

  16. [34]

    embedding space, and each decoder reconstructs the output based on the representation in this joint space

    and same padding consistently across all models. embedding space, and each decoder reconstructs the output based on the representation in this joint space. To construct the Fused V AE, we modified the ELBO: LELBO−qϕ1 = Eqϕ1(z|x) [logpθ1(x|z)] | {z } Reconstruction Loss 1 + Eqϕ...

  17. [36]

    Experiments H.3.1 Artificial neural networks For our experiments, we selected six pre-trained neural networks. Among them, we used three publicly available convolutional neural network (CNN) architectures, VGG16 [Simonyan and Zisserman, 2014], ResNet50 [He et al., 2016], and E...

  18. [38]

    To compute the intra-space cost matrixCXX for neural activity responses, we use a correlation-driven cost similar to Yamins et al. [2014]. Let xi,xj∈X be the neural responses to stimuli si andsj, respectively. We define the cost matrix as: CXX (xi,xj) = 1− cov(xi,xj)p var(xi)v...

  19. [39]

    using support vector machine (SVM) classifiers Chang and Lin 24 X t = 0.0 t = 0.4t = 0.2 t = 1.0t = 0.9t = 0.7t = 0.6 Y Y X Z Source Y X X XX X X Figure 15: Mapping from Swiss roll to Spiral using local alignment Figure 16: Runtime per iteration (in seconds) for different alig...

  20. [40]

    For V4 and IT region, we trained separate SVMs to decode neural activity corresponding to the category of core images using the entire training dataset

    to compute the mappings. For V4 and IT region, we trained separate SVMs to decode neural activity corresponding to the category of core images using the entire training dataset. We employed a C-Support Vector Classification(C-SVC) model with a linear kernel and hinge loss with...

  21. [224]

    For image pre-processing, we followed the procedure outlined in Dosovitskiy et al. [2021]. H.2.2 Sentence Transformer The Sentence Transformer [Reimers and Gurevych, 2019], commonly known as SBERT, converts sentences and paragraphs into embeddings that capture the high-level s...

  22. [1992]

    Geodesic sinkhorn for fast and accurate optimal transport on manifolds

    Guillaume Huguet, Alexander Tong, María Ramos Zapatero, Christopher J Tape, Guy Wolf, and Smita Krishnaswamy. Geodesic sinkhorn for fast and accurate optimal transport on manifolds. In 2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing (MLSP), page...

  23. [2011]

    Linearly mapping from image to text space

    Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162,

  24. [2013]

    Optimal transport tools (ott): A jax toolbox for all things wasserstein

    Marco Cuturi, Laetitia Meng-Papaxanthos, Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul. Optimal transport tools (ott): A jax toolbox for all things wasserstein. arXiv preprint arXiv:2201.12324,

  25. [2014]

    Gera: Label-efficient geometrically regularized alignment

    Dustin Klebe, Tal Shnitzer, Mikhail Yurochkin, Leonid Karlinsky, and Justin Solomon. Gera: Label-efficient geometrically regularized alignment. arXiv preprint arXiv:2310.00672,

  26. [2015]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747,

  27. [2017]

    A contribution to optimal transport on incomparable spaces

    Titouan Vayer. A contribution to optimal transport on incomparable spaces. arXiv preprint arXiv:2011.04447,

  28. [2018]

    Tom B Brown

    URL http://github.com/google/jax. Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165,

  29. [2019]

    Minibatch optimal transport distances; analysis and applications

    Kilian Fatras, Younes Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Minibatch optimal transport distances; analysis and applications. arXiv preprint arXiv:2101.01792,

  30. [2021]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11

  31. [2022]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee,

  32. [2023]

    The platonic representation hypothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987,

  33. [2024]

    Learning with minibatch wasserstein: asymptotic and gradient properties

    10 Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein: asymptotic and gradient properties. arXiv preprint arXiv:1910.04091,

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.