{"id":"a2527182-4f4f-4d25-8e4a-815351510c12","arxiv_id":"2505.12322","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A semi-supervised conditional flow matching method, guided by a new inter-modal bridge cost, aligns latent spaces of distinct pre-trained models using only a small set of paired samples.","lead":"This paper proposes a way to connect two unrelated AI models, like a vision model and a language model, by learning a smooth mapping between their internal representations using only a small number of paired examples. It matters because it could make it cheaper and easier to reuse powerful existing models across different data types, such as images, text, or brain recordings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bridge-cost evaluation in Fig. 6 appears circular: Cbridge assigns zero cost to paired anchors and matching accuracy is scored on the same paired set, so the global-alignment branch of the central claim may rest on an artifact.","rationale":"In good faith, the paper's core proposal is a two-setting method, and the bridge cost is the sole new component that separates global alignment from directly using paired samples. The downstream experiments are plausible, but the bridge cost's only dedicated validation is Fig. 6, whose caption does not rule out leakage from anchors into the scoring set. The reader's stated weakest assumption (feature extractor quality) is real but is explicitly acknowledged in Sec. 5.1.2 and the conclusion; it qualifies the method's scope without threatening the internal logic of the experiments. The circularity concern, by contrast, would invalidate the evidence for a specific claimed contribution. I would not reject: true alignment does not depend on the matching-accuracy comparison, and the ImageNet/MNIST downstream curves could survive even if the bridge-cost advantage is overstated. But the global-alignment half of the central claim should be treated as conditional on a non-circular evaluation, and the paper should state exactly which pairs were used to define the cost versus to compute accuracy. A single held-out split test would settle the question.","tokens_in":26191,"tokens_out":7819,"duration_ms":83805,"concrete_test":"Recompute the matching-accuracy results in Fig. 6 and App. L with a strict split: build Cbridge (and the KNN/KCCA costs) using only a training anchor set P_train, solve the OT coupling on a separate set of source/target features, and score matches only on pairs where neither element is in P_train; equivalently, report a memorization-vs-generalization breakdown by excluding the anchor rows and columns from the coupling before scoring. If the bridge cost no longer outperforms KNN/KCCA on held-out pairs, then the global-alignment results in Figs. 8 and 9 should be re-examined and the abstract claim should be restricted to true alignment.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical justification for the new inter-modal bridge cost is Fig. 6 (and App. L), which reports matching accuracy of the optimal coupling obtained with bridge, KNN, and KCCA costs. The bridge cost Cbridge in Eq. 4 is constructed by assigning zero cost to every pair in P, and the figure caption states that matching accuracy is computed by sampling from the coupling pi_star and checking how often the matched items have the same label. If the pairs in P used to build the cost are the same pairs scored in this evaluation, the comparison is partly circular: the OT solver will concentrate mass on the zero-cost anchor pairs, and because paired anchors are label-consistent by construction, those samples count as correct regardless of whether the bridge cost generalizes to unseen pairs. The paper does not report excluding anchor pairs or using a held-out paired set for scoring. This matters because the claimed superiority over KNN/KCCA is the stated reason the bridge cost is adopted in all subsequent global-alignment experiments; if that advantage disappears under a non-circular evaluation, the global-alignment branch of the central claim ('under both settings') no longer has independent support. The true-alignment results are not affected by this issue, so the paper would remain conditionally viable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a semi-supervised method for aligning latent spaces of pre-trained models across modalities, using conditional flow matching. Paired samples are used in two ways: as direct anchors for 'true' alignment, or as zero-cost bridges in a novel inter-modal cost matrix that is fed into an optimal transport solver to obtain a 'global' coupling. The learned flow is then used to morph between latent spaces, and the method is evaluated on image-text alignment (MNIST, ImageNet) and on aligning artificial neural network representations to neural recordings (Majaj et al., 2015). The central claim is that, with less than 20% paired data, the method matches the downstream task performance of end-to-end trained models on object recognition and image generation.","tokens_in":26374,"tokens_out":7178,"duration_ms":70547,"significance":"If the central claim holds, the paper offers a practical, lightweight route to reusing pre-trained encoders across modalities with very little paired data, and its biological-to-artificial alignment results would be a useful contribution to the neural-predictivity literature. The paper's strengths are its broad benchmark coverage, five-seed experiments, and explicit ablations of cost functions, OT solvers, and velocity-field architectures. The feature-overlap analysis in Sec. 5.1.2 and Fig. 5 also makes a falsifiable prediction connecting representation quality to morph quality. However, the global-alignment branch currently rests on a partly circular evaluation and on unspecified large-scale OT details, so the headline claim is stronger than the evidence.","major_comments":[{"comment":"The evaluation of the bridge cost appears circular. Eq. (4) assigns zero cost to every pair in the anchor set P, and Fig. 6 reports matching accuracy by sampling from the optimal coupling and checking label agreement. Nothing in the caption or text indicates that the anchor pairs used to build the cost are excluded or held out when computing this accuracy. Because anchor pairs are label-consistent by construction, an OT solver will concentrate mass on them, so a high matching score may reflect the cost construction rather than generalization to unseen pairs. Please re-run the comparison in Fig. 6 with a held-out paired set (or by scoring only non-anchor pairs), and report both anchor and non-anchor accuracy. If the bridge advantage over KNN/KCCA disappears, the global-alignment branch of the abstract's 'under both settings' claim no longer has independent support.","section":"§5.1.3, Fig. 6; App. L"},{"comment":"The central claim that the method 'matches downstream task performance ... particularly when labelled training data is scarce (<20%)' is only partially supported by the ImageNet results. The text states that local alignment failed to converge and that the global solver underperformed with very few paired samples, improving to a level comparable to the classifier only at roughly 10% paired data. The figure and text also do not report the true-alignment curve for ImageNet separately, even though 'both settings' is claimed. Please report the true-alignment ImageNet results explicitly, or revise the claim to state which settings and data fractions are actually supported.","section":"§5.1.5, Fig. 8"},{"comment":"Eq. (4) is not well-defined as written. The anchor set P contains pairs of the form (x^p_i, y^p_j), but the second line of the minimum is written over '(x^p_i, x^p_j) ∈ P' and then uses C_YY(y^p_j, y_j), mixing source and target indices. This should be rewritten as a minimum over anchor indices k of C_XX(x_i, x^p_k) + C_YY(y^p_k, y_j). Since this equation defines the proposed bridge cost, the notation needs to be corrected before the method can be implemented or reproduced.","section":"§3.3, Eq. (4)"},{"comment":"The scalability of the global-alignment experiments is not specified. The text says global alignment computes an OT plan over the full dataset, but the ImageNet training split is 50% of roughly 1.2M images (about 600K points), for which a dense pairwise cost matrix is infeasible. Please state the exact number of points used for the global OT solve in Fig. 8, the subsampling procedure, and the discrete solver (e.g., Sinkhorn on a subsample, or a low-rank method). This information is necessary to reproduce the reported results and to assess the scalability claims.","section":"§5.1.5, App. E"}],"minor_comments":[{"comment":"The contributions list contains two items labelled '(b)' and no '(c)'; please renumber the list.","section":"Section 1"},{"comment":"The pseudocode computes πtrue for true alignment and then unconditionally overwrites π with an OT coupling, so it does not reflect the described three strategies; add explicit conditional branches for true, global, and local alignment.","section":"Algorithm 1, App. E"},{"comment":"The sentence 'we assessed ... (Sec 5.1.3) (Sec 5.1.4)' contains a duplicated parenthetical reference; please clean up the cross-references.","section":"§5.1.3"},{"comment":"The 'Paired:Unpaired' axis labels are not defined in the main text or captions; please state whether the values are fractions of the full training set and define the exact ratios used.","section":"Figs. 6, 8, 9"},{"comment":"The initials 'U-GENOT' and 'GENOT' are used interchangeably; please use one consistent name for the base method.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern in Fig. 6 is the main risk: if the bridge-cost advantage is not reproduced on held-out anchors, the global-alignment branch should be de-emphasized or removed from the central claim. The ImageNet scalability details and the missing true-alignment curve should also be verified before acceptance. The paper is within the scope of cs.LG and the topic is timely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical paper on aligning pre-trained model latent spaces with few paired samples. The true-alignment results are solid and the comparison of true/global/local strategies is worth having. But the headline advantage of the new bridge cost over KNN/KCCA rests on an in-sample evaluation that is partly circular, so treat the global-alignment branch with caution.\n\nThe new thing is the bridge cost (Eq. 4): paired samples are zero-cost anchors and any other pair's cost is the shortest path through the two intra-space costs. That is a natural extension of fused-cost ideas, and the paper gives a clean comparison against KNN and KCCA costs, plus a systematic study of true vs global vs local alignment. The downstream experiments on MNIST, ImageNet, and the Majaj neural data are reasonably thorough.\n\nThe main problem is Fig. 6. The bridge cost assigns zero cost to the paired anchors, and the matching accuracy is scored by sampling from the same optimal coupling built from those anchors. Since the anchors are label-consistent, the solver will put mass on them and those samples count as correct regardless of generalization. The paper never says the scoring used a held-out paired set. That makes the claimed superiority of the bridge cost partly a tautology. If the advantage disappears on held-out pairs, the justification for using the bridge cost in the global-alignment experiments weakens. The true-alignment results stand on their own, so the paper is not sunk, but the abstract's 'under both settings' claim is only partially supported.\n\nAlso, the ImageNet downstream result (Fig. 8) shows global alignment roughly matching the ViT classifier baseline at 10% pairs, not clearly exceeding it; the claim is 'matches', so that is okay, but local alignment failing to converge under the 18-hour budget is a real limitation. Hyperparameter tuning is on the validation set and no code is shipped, which are minor reproducibility issues.\n\nWho is this for? People working on model stitching, cross-modal transfer, or low-supervision alignment. It is a decent empirical contribution, but the central new cost function needs a cleaner evaluation before I'd rely on it.\n\nI'd send it to peer review. The method is reasonable, the experiments are broad, and the circularity can be fixed by a held-out evaluation. A serious referee should ask for that.","headline":"Useful empirical study of alignment strategies, but the bridge-cost advantage rests on a circular evaluation.","tokens_in":26973,"tokens_out":3749,"would_cite":false,"duration_ms":35321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a flow trained on fewer than 20% paired samples can fuse pre-trained models across modalities — vision, text, and brain recordings — matching end-to-end accuracy on recognition and generation.","keywords":["model alignment","conditional flow matching","inter-modal bridge cost","optimal transport","semi-supervised learning","latent space alignment","multimodal representation learning","neural representation alignment"],"falsifier":"Take a source encoder whose latent space is deliberately scrambled (for example, a vision transformer with class tokens randomly permuted) so within-space distances no longer track class identity, keep the same paired anchors, and run the full pipeline; if coupling accuracy and downstream accuracy do not collapse, the intra-space geometry is not what carries the alignment. A complementary check on the same principle: on a dataset with full labels, compute the bridge-cost coupling from a random 10% anchor set and count how many of the highest-weight coupling entries match ground-truth correspondences, comparing against the fused costs the paper benchmarks in its Figure 6.","tokens_in":25913,"feed_emoji":"🔗","tokens_out":14081,"duration_ms":121616,"temperature":0.7,"pith_summary":"The paper claims that two pre-trained models living in different modalities — a vision encoder and a text encoder, or a deep network and a patch of monkey visual cortex — can be fused by a single learned flow between their latent spaces, and that this flow needs fewer than 20% of samples to be paired in order to match the downstream accuracy of models trained end-to-end on both modalities. The alignment is computed in one of two ways: directly from the labelled pairs, or by solving an optimal transport problem whose cost matrix is built from a new bridge cost that treats the sparse paired samples as zero-cost anchors and expresses every cross-space distance as a chain of within-space distances. The paper reports parity with end-to-end baselines on MNIST image-to-text and text-to-image tasks, on ImageNet image-to-text classification measured against the vision transformer's own classification head, and on category decoding from mapped neural activity. The reason this would matter is modularity: if the claim holds, frozen feature extractors can be coupled cheaply, without retraining either model, whenever a small anchor set is available.","feed_headline":"Under 20% paired data is enough to bridge vision and text models","feed_subtitle":"Frozen encoders, vision to text to brain, fuse via a lightweight flow at end-to-end accuracy.","key_machinery":"The load-bearing object is the inter-modal bridge cost, a rule that converts sparse labels into a dense cross-space distance matrix: $C^{XY}_{\\text{bridge}}(x_i, y_j) = 0$ when the pair is a labelled anchor, and otherwise $C^{XY}_{\\text{bridge}}(x_i, y_j) = \\min_{(x^p_i, y^p_j) \\in P} \\big[ C_{XX}(x_i, x^p_i) + C_{YY}(y^p_j, y_j) \\big]$, with within-space costs taken as cosine distances (or correlation-based costs for neural data). The work this cost does is to make two incomparable latent spaces comparable so an entropic optimal transport solver — linear, fused Gromov–Wasserstein, or unbalanced — can produce a coupling $\\pi^*$; conditional flow matching then trains the velocity field $v_{t,\\theta}$ against the conditional vector field $u_t(x \\mid x_1)$, and out-of-sample predictions come from solving the induced ODE from noise to target conditioned on the source point. The velocity field is parameterised with adaptive-normalisation blocks conditioned on time and source input, which the paper shows learn faster and reach higher accuracy than the plain multi-layer-perceptron architecture of the cross-domain flow-matching method it builds on.","core_discovery":"The paper's central claim is that inter-modal alignment can be decoupled from representation learning: once each side has a frozen pre-trained encoder, the correspondence between their latent spaces is itself a learnable object. Its vehicle is a new inter-modal bridge cost $C^{XY}_{\\text{bridge}}(x_i, y_j)$ over pairs from two spaces that share no common metric: paired samples $(x^p_i, y^p_j)$ get cost zero and act as anchors, and any other pair is charged the minimum over anchors of the within-space distance from $x_i$ to its anchor plus the within-space distance from the anchor's partner to $y_j$. This dense cost matrix feeds a balanced or unbalanced entropic optimal transport solver whose coupling $\\pi^*$ defines the training pairs for a conditional flow: a time-dependent velocity field $v_{t,\\theta}(y \\mid x)$ is trained to morph noise into the target distribution conditioned on a source point, giving out-of-sample transport by solving an ODE. With under 20% of samples paired, the paper reports downstream accuracy matching end-to-end trained models on object recognition across MNIST and ImageNet and on MNIST image generation, and it reports that on noisy biological data the global transport alignment improves as pairing grows while direct paired alignment overfits.","pith_inferences":["A natural stress test the paper leaves implicit: connect two encoders never seen together (for example, a newly released vision encoder to an existing text encoder) using a few hundred anchors and check whether the same hyperparameters transfer; if the method is truly modular, they should.","As the number of anchors grows, the bridge cost should approach the geodesic-style fused cost built from k-nearest-neighbour graphs in the appendix; verifying that downstream accuracy saturates smoothly with anchor count, rather than requiring a minimum threshold, would pin down the cost's scaling behaviour.","The under-20% claim assumes anchors are sampled at random, while real deployments often pair only easy classes; a testable extension is to measure accuracy when anchors cover only a subset of classes, since unseen classes would then be routed entirely through within-space distances — exactly the regime the min-over-anchors construction is designed for.","The measured link between source-space overlap and flow quality suggests a selection rule the paper does not state: when several pre-trained encoders are available for a modality, use the one with the most separated latent classes as the source; the six-network neural experiment already contains the data to rank encoders and check this against test error."],"forward_implications":["With fewer than 20% of samples paired, the learned flow matches end-to-end trained baselines on MNIST image-to-text and text-to-image tasks, and outperforms them below about 10% pairing.","On ImageNet, image-to-text accuracy rises with the paired fraction and reaches a level comparable to the vision transformer's own classification head at roughly 10% pairing, while per-batch local alignment fails to converge within the 18-hour training budget.","On biological-to-artificial alignment, global transport alignment's test error falls as pairing grows, whereas direct paired alignment overfits with more pairs — favouring the optimal-transport route on noisy neural data.","The bridge cost yields higher coupling matching accuracy than fused costs built from k-nearest-neighbour graphs or kernel canonical correlation analysis, with the best discrete solver (fused Gromov–Wasserstein on MNIST, linear on ImageNet) depending on the dataset.","Category decoding from activations mapped to IT and V4 tracks Brain-Score rankings, and the layer-to-region pattern — later layers fit IT, early-to-middle layers fit V4 — matches the ventral-stream hierarchy."],"supporting_citations":[{"why":"Supplies the cross-domain generative entropic optimal transport framework this paper extends, including its per-batch local alignment baseline and neural reweighting terms.","marker":"[Klein et al., 2023]"},{"why":"Defines conditional flow matching, the objective used to train the velocity field in Eq. 1.","marker":"[Lipman et al., 2022]"},{"why":"Provides the entropic optimal transport formulations and solver background used to compute the couplings.","marker":"[Peyré and Cuturi, 2019]"},{"why":"Gives the Sinkhorn algorithm that computes the entropic optimal transport plans.","marker":"[Cuturi, 2013]"},{"why":"Supplies the ViT-Base image encoder whose features are aligned to text, and the 83.97% classification-head accuracy used as the ImageNet ceiling.","marker":"[Dosovitskiy et al., 2021]"},{"why":"Supplies the MiniLM-L6 sentence encoder that produces the text-latent targets for the ImageNet alignment.","marker":"[Wang et al., 2020]"},{"why":"Provides the monkey V4/IT neural recording dataset and the SVM decoding protocol used to evaluate biological-artificial alignment.","marker":"[Majaj et al., 2015]"},{"why":"Supplies the Brain-Score benchmark to which the neural-alignment results are compared.","marker":"[Schrimpf et al., 2018]"},{"why":"Supplies the VAE architecture for the MNIST image and text latent spaces and for the end-to-end fusion baselines.","marker":"[Kingma and Welling, 2014]"}],"fun_headline_variants":["Frozen encoders align with 20% paired data via bridges","Bridge cost aligns models across modalities with <20% pairs","Semi-supervised flow matching aligns vision and text models","Inter-modal bridges align models with minimal supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the pre-trained feature extractors doing their job: if the source latent space does not separate classes by distance — the paper measures this as feature-space overlap and shows performance declining as overlap grows — the bridge cost and the learned flow carry no reliable signal, so alignment quality is bounded by the upstream models' capacity.","fun_headline_variants_meta":{"raw":{"variants":["Frozen encoders align with 20% paired data via bridges","Bridge cost aligns models across modalities with <20% pairs","Semi-supervised flow matching aligns vision and text models","Inter-modal bridges align models with minimal supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000972,"raw_usage":{"total_tokens":4153,"prompt_tokens":988,"completion_tokens":3165,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":3099}},"tokens_in":604,"tokens_out":3165,"duration_ms":20845,"temperature":1.0,"reasoning_tokens":3099,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:35:27.678858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a source encoder whose latent space is deliberately scrambled (for example, a vision transformer with class tokens randomly permuted) so within-space distances no longer track class identity, keep the same paired anchors, and run the full pipeline; if coupling accuracy and downstream accuracy do not collapse, the intra-space geometry is not what carries the alignment. A complementary check on the same principle: on a dataset with full labels, compute the bridge-cost coupling from a random 10% anchor set and count how many of the highest-weight coupling entries match ground-truth correspondences, comparing against the fused costs the paper benchmarks in its Figure 6.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ViT-Base image encoder whose features are aligned to text, and the 83.97% classification-head accuracy used as the ImageNet ceiling."}],"review_version":1}