{"id":"a889918c-0e1c-4571-a167-b86b5dd79379","arxiv_id":"2411.09723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A contrastive model aligns EEG, MEG, and fMRI activity to CLIP image embeddings, enabling image retrieval from brain signals, neural retrieval from images, and cross-modal neural retrieval.","lead":"This paper trains separate EEG, MEG, and fMRI models to map brain recordings into the same image-representation space used by the CLIP model, then retrieves matching images, neural patterns, or other-modality patterns. The idea matters because a single shared space could let researchers translate between different brain-scanning techniques and build more general brain-decoding tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Modality conversion results may be explained by transitivity through CLIP image similarity rather than by a learned cross-modal neural alignment; the paper provides no control separating these.","rationale":"The reader's verdict already identifies the key training setup in its weakest_assumption: each modality is trained independently against CLIP image embeddings, so cross-modal alignment is not guaranteed. My stress-test pushes one step further: even in the best case where each encoder perfectly maps to CLIP space and the space is dense, the modality conversion experiment is structurally unable to demonstrate cross-modal neural alignment, because the query and target are connected only through the shared image anchor. The reported conversion numbers are therefore compatible with a much weaker claim: each modality independently approximates CLIP image embeddings, and conversion is transitive CLIP similarity. This does not make the paper's results wrong, and the decoding results (e.g., top1 40% for EEG, fMRI 93.8% CLIP 2-way) are plausible and in line with prior work, so I do not see a reason to reject. However, the interpretation as a 'foundation model' or a unified neural representation space needs a control that rules out the transitive CLIP baseline. The proposed oracle test (querying with the source image's CLIP embedding) is a clean, inexpensive way to set an upper bound; if the neural-query accuracy is close to it, the conversion experiment adds no evidence of neural-to-neural alignment beyond image semantics. The absence of overlapping stimuli across the three datasets reinforces this: exact cross-modal correspondence is never measured. I therefore agree with the conditional verdict, with the condition explicitly including this baseline and a direct statement about the disjoint stimulus sets.","tokens_in":12005,"tokens_out":9296,"duration_ms":93026,"concrete_test":"For each conversion direction in Table 2, add a transitive baseline that replaces the source neural encoder with the CLIP image encoder applied to the source image (i.e., query with h(img_i) instead of f_X(n_X)), keeping the same target retrieval set and CLIP 2-way metric. If the neural-query conversion accuracy is not significantly above this image-only baseline, the conversion result is fully accounted for by CLIP image similarity and does not demonstrate cross-modal neural alignment. Additionally, check whether any images overlap between the source and target retrieval sets; if none exist, state this explicitly and weaken the 'shared neural space' claim to 'shared semantic anchoring via CLIP.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is in Sections 2.2–2.4: each neural encoder f_n is trained separately with the CLIP contrastive loss against the same frozen image encoder h, with no loss term pairing different neural modalities. Modality conversion (Section 2.3) is then implemented as nearest-neighbor retrieval: given f_X(n_X) for image i, retrieve n_Y maximizing cosine(f_X(n_X), f_Y(n_Y)). Because f_X(n_X) approximates h(img_i) and f_Y(n_Y) approximates h(img_j), this is approximately argmax_j h(img_i)·h(img_j) — i.e., plain CLIP image similarity. The evaluation metric in Section 2.4 is CLIP 2-way accuracy between the images associated with the source and retrieved target samples, which measures exactly that image-level semantic similarity. Therefore the high conversion accuracies in Table 2 (e.g., normalized 0.954 for fMRI→MEG) are expected under the null model that each modality independently decodes to CLIP space; they do not provide evidence for a shared neural representational space or for cross-modal neural correspondence. The paper's normalization of conversion accuracy by target-modality decoding accuracy does not disentangle this image-anchor transitivity. The absence of any test with identical stimuli across modalities (the EEG, MEG, and fMRI datasets have disjoint image sets) means the core claim of cross-modal neural alignment is never directly tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework that trains separate EEG, MEG, and fMRI encoders with a contrastive CLIP loss against a frozen CLIP image encoder, and then uses nearest-neighbor retrieval in the resulting shared space for three tasks: decoding images from neural data, encoding images into neural data, and converting between neural modalities. The reported results include CLIP 2-way decoding accuracies of 79.4% for EEG, 60.1% for MEG, and 93.8% for fMRI, encoding accuracies between 58.8% and 87.8%, and normalized modality-conversion accuracies between 83.7% and 95.4%. The central claim is that this demonstrates a unified semantic representation shared across EEG, MEG, and fMRI. The manuscript is a proof-of-concept study built on public datasets and includes a useful comparison with prior decoding work, but the evidence for the modality-conversion claim is incomplete.","tokens_in":12252,"tokens_out":3703,"duration_ms":38589,"significance":"If the modality-conversion results were properly supported, this would be a valuable step toward a unified multimodal neural representation for vision: one framework handling decoding, encoding, and cross-modal retrieval, with public datasets and attention to known EEG contamination pitfalls. The paper is honest about its proof-of-concept status and non-generative nature. However, the load-bearing conversion claim currently lacks a control separating genuine cross-modal neural alignment from transitivity through CLIP image similarity, and the reported numbers are single point estimates without variability or a defined normalization procedure. The significance is therefore conditional on additional analysis.","major_comments":[{"comment":"","section":"Sections 2.2-2.3, Table 2"},{"comment":"","section":"Section 2.4, Table 2"},{"comment":"","section":"Tables 1-2, Section 3"},{"comment":"","section":"Section 2.4, Tables 1-2"}],"minor_comments":[{"comment":"","section":"Abstract and Section 1"},{"comment":"","section":"Table 1"},{"comment":"","section":"Section 2.1"},{"comment":"","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a vision or neuroimaging journal, but the core novelty is the cross-modal conversion claim, and that claim currently rests on a confounded evaluation. I would be willing to revisit after the authors add the transitivity control and reporting of variability. No concerns about citation practices; the related work coverage is adequate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper trains separate EEG, MEG, and fMRI encoders to match frozen CLIP image embeddings, and gets decoding numbers that line up with prior single-modality work. The genuinely new hook—cross-modal 'modality conversion'—is not supported by the evidence as presented. The conversion experiment is nearest-neighbor retrieval between independently trained encoders, and the metric used to score it (CLIP 2-way accuracy between the images tied to the source and retrieved neural samples) is exactly the quantity that should be high under the null hypothesis that each encoder independently recovers CLIP space. You don't need any shared neural alignment to get those numbers; you just need CLIP image similarity to be transitive. So Table 2 is not evidence for a unified neural representation.\n\nWhat the paper does well: it's a clean, readable application of contrastive alignment to three public datasets, it carefully avoids the known EEG train/test contamination problem, and the per-modality decoding results look consistent with contamination-aware baselines (EEG top1 40%, MEG top5 6.1%, fMRI CLIP 2-way 93.8%). The encoding direction (image-to-neural retrieval) is a nice addition, and the authors are explicit that this is a retrieval-only proof of concept.\n\nThe soft spots: the conversion argument has no control. Because EEG, MEG, and fMRI come from different subjects and disjoint image sets, the paper never tests whether activity from the same stimulus lands near itself across modalities. The normalization in Table 2 (dividing by target-modality decoding accuracy) does not address transitivity; it just rescales. Also missing: error bars or subject-level variability, the exact data splits, and the code. Some of this is minor (code would help reproducibility), but the transitivity issue is load-bearing.\n\nMy verdict: the decoding/encoding results are worth a serious look, and the paper is a fair baseline for future work. But the cross-modal claim needs a direct control—for instance, comparing modality-conversion accuracy against plain CLIP image-to-image retrieval. That control would determine whether 'conversion' is a real neural phenomenon or a byproduct of the shared anchor. I'd send this to peer review, because the authors know the literature and the framework is reusable, but I'd expect reviewers to demand that control, along with error bars and code, before acceptance.","headline":"Solid CLIP-anchored decoding for EEG/MEG/fMRI, but the modality-conversion headline rests on an evaluation that measures CLIP image similarity, not a learned cross-modal neural alignment.","tokens_in":12806,"tokens_out":3561,"would_cite":false,"duration_ms":34399,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that EEG, MEG, and fMRI encoders trained with contrastive learning against frozen CLIP image embeddings create a shared semantic space that supports decoding, encoding, and cross-modality conversion by nearest-neighbor…","keywords":["brain decoding","brain encoding","neural modality conversion","contrastive learning","electroencephalography","magnetoencephalography","functional magnetic resonance imaging","shared representation space"],"falsifier":"Run modality conversion on a held-out set after training the EEG and fMRI encoders on disjoint image sets, then check whether the CLIP 2-way conversion accuracy stays above chance; if it stays high, the retrieval is carried by image-level CLIP similarity in the candidate set rather than by an aligned neural code.","tokens_in":11785,"feed_emoji":"🧠","tokens_out":5345,"duration_ms":45412,"temperature":0.7,"pith_summary":"This paper tries to show that three very different ways of measuring brain activity—electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI)—can be pulled into one shared semantic space anchored by the frozen image encoder of the CLIP model. The authors train a separate neural encoder for each modality with a contrastive loss that pushes the encoder's output toward the CLIP embedding of the image the subject was viewing. In that shared space, all three tasks reduce to nearest-neighbor retrieval: decoding (find images matching a brain signal), encoding (find brain signals matching an image), and modality conversion (find EEG/MEG/fMRI activity that matches activity recorded in another modality). The paper reports 93.8 percent CLIP 2-way accuracy for fMRI decoding, 79.4 percent for EEG, 60.1 percent for MEG, and normalized conversion accuracies from 83.7 to 95.4 percent. If these numbers hold under clean evaluation splits, the framework offers a single non-generative backbone for aligning neural recordings to visual content and to each other.","feed_headline":"One shared space aligns EEG, MEG, and fMRI to images","feed_subtitle":"Decoding, encoding, and cross-modal conversion all run as nearest-neighbor retrieval in one CLIP-anchored space.","key_machinery":"The central object is the shared CLIP-anchored representation space. CLIP's frozen image encoder $h$ produces a semantic embedding for every image; each neural modality gets a trainable module $f_n$ that maps raw signals into the same normalized unit hypersphere. The contrastive loss—symmetric cross-entropy with a temperature of $\\tau = 1$ between the neural embedding and the CLIP image embedding for the matching image—is what forces the alignment. This single mechanism carries all three experiments: because every modality is anchored to the same image embeddings, nearest-neighbor search in that space implements decoding, encoding, and cross-modal conversion without any direct pairwise training between modalities.","core_discovery":"On its own terms, the paper's central claim is that aligning each neural modality to CLIP image embeddings is enough to create a joint representation space shared across modalities, without any EEG-MEG or EEG-fMRI training pairs. Each neural module $f_n = g_n \\circ a_n$ first maps a subject's raw signal into a common per-subject-aligned space and then refines it to approximate the CLIP image representation $h(\\mathrm{img})$. After training, decoding, encoding, and modality conversion are all implemented as cosine-similarity nearest-neighbor search in that space, and the paper evaluates all three with CLIP 2-way accuracy between the images associated with query and retrieved samples. The authors report that fMRI decoding reaches 93.8 percent, EEG 79.4 percent, and MEG 60.1 percent CLIP 2-way accuracy, while conversion accuracies normalized by decoding performance range from 83.7 to 95.4 percent. They explicitly note the model is non-generative and proof-of-concept.","pith_inferences":["Inference: If the shared space is as dense as the reported accuracies suggest, modality conversion could be used to impute one recording type from another (for example, estimating fMRI-like spatial detail from cheaper EEG recordings) without retraining, though the paper does not test this directly.","Inference: The reliance on CLIP as the anchor means the semantic granularity of the model is bounded by CLIP's own representations; images that CLIP cannot separate will likely not be separable in any modality, and that ceiling should be measurable by computing CLIP 2-way accuracy on the image retrieval set itself.","Inference: A direct stress test of the shared-space claim would be to train the EEG and fMRI encoders on disjoint image sets and then measure cross-modal conversion on images seen by only one modality; high accuracy there would show genuine neural correspondence rather than image-level similarity leakage.","Inference: The paper's privacy discussion implies that as such decoders improve, the same alignment could be used for cross-subject transfer or minimal-cooperation decoding, making neural-privacy safeguards more urgent; the authors raise but do not resolve this concern."],"forward_implications":["A single framework can retrieve the image a person was viewing from EEG, MEG, or fMRI, with reported CLIP 2-way decoding accuracies of 79.4 percent, 60.1 percent, and 93.8 percent, respectively.","The same encoders run encoding in reverse: given an image, they retrieve neural activity whose associated image is semantically similar, with reported CLIP 2-way accuracies of 85.5 percent (EEG), 58.8 percent (MEG), and 87.8 percent (fMRI).","Neural modality conversion is possible without cross-modal training pairs: for example, fMRI-to-MEG conversion reaches 95.4 percent normalized CLIP 2-way accuracy by nearest-neighbor search in the shared space.","All three tasks use identical machinery (contrastive alignment followed by cosine-similarity retrieval), so improvements in the alignment step should transfer across tasks.","Because the framework is non-generative, decoded or converted outputs are retrieved existing images or neural samples rather than newly generated signals; the authors position the model as a proof-of-concept rather than a full foundation model."],"supporting_citations":[{"why":"Supplies the frozen CLIP image encoder whose embeddings are the semantic anchor for all neural modules.","marker":"[1]"},{"why":"Provides the ImageNetEEG dataset used for EEG training and evaluation.","marker":"[29]"},{"why":"Provides the THINGS-MEG dataset used for MEG training and evaluation.","marker":"[32]"},{"why":"Provides the Natural Scenes Dataset used for fMRI training and evaluation.","marker":"[34]"},{"why":"Establishes the contrastive fMRI-decoding approach and the CLIP 2-way accuracy metric the paper compares against.","marker":"[24]"},{"why":"Supplies the conservative block-design EEG splitting protocol used to avoid train/test contamination.","marker":"[28]"},{"why":"Documents the train/test contamination in earlier EEG decoding that motivates the clean evaluation split.","marker":"[31]"},{"why":"Provides the MEG decoding baseline on THINGS data the paper compares its MEG results to.","marker":"[39]"}],"fun_headline_variants":["CLIP anchors a shared brain-activity space for EEG, MEG, fMRI","One contrastive space unifies EEG, MEG, fMRI with images","Decode, encode, convert: CLIP-aligned neural modalities","Brain-to-image retrieval in a CLIP-shared space","EEG, MEG, fMRI all speak CLIP in one space"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the shared CLIP-anchored space is dense and semantically consistent enough that nearest-neighbor retrieval from one modality lands on genuinely matching neural activity in another modality, a property the contrastive objective does not by itself guarantee.","fun_headline_variants_meta":{"raw":{"variants":["CLIP anchors a shared brain-activity space for EEG, MEG, fMRI","One contrastive space unifies EEG, MEG, fMRI with images","Decode, encode, convert: CLIP-aligned neural modalities","Brain-to-image retrieval in a CLIP-shared space","EEG, MEG, fMRI all speak CLIP in one space"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3661,"prompt_tokens":862,"completion_tokens":2799,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":2705}},"tokens_in":478,"tokens_out":2799,"duration_ms":18965,"temperature":1.0,"reasoning_tokens":2705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:40:19.459504+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run modality conversion on a held-out set after training the EEG and fMRI encoders on disjoint image sets, then check whether the CLIP 2-way conversion accuracy stays above chance; if it stays high, the retrieval is carried by image-level CLIP similarity in the candidate set rather than by an aligned neural code.","supporting_citations":[{"cited_title":"Spampinato, S","cited_arxiv_id":null,"evidence_quote":"Provides the ImageNetEEG dataset used for EEG training and evaluation."},{"cited_title":"Correct block-design experiments mitigate temporal correlation bias in EEG classification","cited_arxiv_id":"2012.03849","evidence_quote":"Supplies the conservative block-design EEG splitting protocol used to avoid train/test contamination."},{"cited_title":"Training on the test set? An analysis of Spampinato et al. [31]","cited_arxiv_id":"1812.07697","evidence_quote":"Documents the train/test contamination in earlier EEG decoding that motivates the clean evaluation split."}],"review_version":1}