{"id":"5633a015-a07f-4506-b11c-2e26f19a59d6","arxiv_id":"2507.12009","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"An encoder-decoder CNN predicts visual-cortex fMRI from movie frames and reconstructs frames from the predicted fMRI, with saliency maps pointing to occipital and fusiform regions.","lead":"This paper trains a convolutional encoder-decoder to predict fMRI responses to movie clips and to reconstruct film frames from that predicted activity. If the method holds up, it offers a tool for studying visual processing under naturalistic conditions, but the reported decoding is computed from model-predicted, not measured, brain activity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decoder is trained and evaluated only on encoder-predicted fMRI, so the reported 'reconstruction from neural activity' may be an autoencoder of movie frames rather than a readout of measured brain activity; a ground-truth fMRI decoding test is needed.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing issue: the decoder is trained and evaluated on encoder-predicted fMRI rather than on measured neural activity. If the decoder cannot operate on ground-truth fMRI, the paper's abstract and Section III-B overclaim by calling the reconstructions 'from neural activity.' This is a correctness risk, not a stylistic disagreement. The paper's other concerns, such as possible leakage from SNR-based voxel selection computed before the split (Section II-A) and non-independence of per-voxel statistics on group-averaged data, are real but secondary; the decoder-input problem alone is sufficient to invalidate the central decoding claim. The paper is clearly written, and the encoder-only results and saliency analysis may be useful ingredients, but the current evidence does not support the stated reconstruction claim. No adjustment to the reader's REJECT verdict is needed.","tokens_in":7518,"tokens_out":2669,"duration_ms":33957,"concrete_test":"Train a decoder with the same architecture and loss from scratch on ground-truth subject-averaged fMRI volumes from the training split (input v, not vhat), with the same middle-frame targets, and evaluate it on held-out ground-truth test fMRI volumes. Compare its SSIM/MSE to the current vhat-based decoder result and to the shuffled-frame null distribution in Fig. 3. As a supplementary probe, run the existing trained decoder on held-out ground-truth fMRI volumes to check for domain-shift failure. If ground-truth-driven SSIM is near the null or substantially below the vhat-based result, the reconstruction-from-neural-activity claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central decoding claim (Abstract; Section III-B) is that the decoder reconstructs movie frames from neural activity. However, in the end-to-end setup the decoder's only input is the encoder's predicted fMRI: Section II-B defines the combined loss in Eq. (3) with the decoder acting on the encoder output, and Section III-B evaluates decoding on 'fMRI data predicted by the encoder' (Fig. 3). Because the encoder's predicted voxel values are a deterministic function of the movie frames, the decoder can learn to invert that movie-derived latent representation without ever reading a measured brain signal. The paper reports no experiment, at train or test time, in which ground-truth fMRI (even subject-averaged) is fed to the decoder. Consequently, the reported SSIM of 0.320 and the saliency maps in Section III-B characterize the encoder-decoder as an autoencoder, not a neural decoding pipeline. This is the load-bearing gap: if the decoder cannot reconstruct frames from measured fMRI, the central claim is unsupported regardless of the encoder's predictive accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end deep encoder-decoder model for naturalistic movie fMRI. The encoder is a temporal CNN that predicts subject-averaged fMRI voxel activity from chunks of 32 film frames; the decoder is a CNN that reconstructs the middle frame from the encoder's predicted fMRI, with a combined loss that includes perceptual, SSIM, and total-variation terms. Using the Emo-FilM dataset, the authors split 13 movies into training/validation and hold out the 14th movie plus the last 20% of the remaining movies as test. They report an average encoding correlation of 0.203 and an average decoding SSIM of 0.320 on the test set, and they use saliency maps to identify middle occipital, fusiform, and calcarine regions as most contributing to reconstruction. The central claims are that the model predicts visual cortex activity and reconstructs the visual stimulus from neural activity, and that the saliency results reflect the brain regions underlying the reconstructed visual features.","tokens_in":7698,"tokens_out":7638,"duration_ms":77981,"significance":"The paper addresses a timely problem—linking naturalistic movie stimuli to fMRI responses—and its encoder-decoder architecture with temporal convolutions is a reasonable design choice. Strengths include the use of the public Emo-FilM dataset, a held-out movie test protocol, null distributions for the encoding and decoding metrics, and non-parametric statistical comparisons between model variants. The saliency analysis yields falsifiable region-level predictions (middle occipital, fusiform, calcarine) that are consistent with established visual cortex functional organization. However, these strengths do not compensate for the fact that the central reconstruction claim is not tested against measured fMRI, and the evaluation hinges on subject-averaged data with a single split. If the authors add a ground-truth fMRI decoding experiment and cross-validated voxel selection, the work could make a solid contribution to naturalistic fMRI encoding/decoding.","major_comments":[{"comment":"The abstract claims that the model performs 'reconstruction of corresponding visual inputs from neural activity,' but the decoder's input is always the encoder's predicted fMRI, not measured fMRI. In the end-to-end setup, the encoder output is a deterministic function of the movie frames, so the decoder can learn to invert that movie-derived latent representation without reading a brain signal. The reported SSIM of 0.320 (Fig. 3) therefore characterizes the encoder-decoder as an autoencoder of movie frames, not as a neural decoding pipeline. The paper does not report any experiment in which ground-truth fMRI (even subject-averaged) is fed to the decoder at train or test time. This point is load-bearing: either the authors must train/evaluate a decoder on measured fMRI, or the decoding claim must be reframed as 'reconstruction from encoder-predicted fMRI,' removing the neural-decoding implications from the abstract and discussion.","section":"Section III-B, Eq. (2)-(3)"},{"comment":"The SNR-based voxel selection is described as a preprocessing step applied to the whole dataset, including the held-out movie and the last 20% of volumes from the remaining movies, before the train/validation/test split (Fig. 1b). Because the top 30% SNR voxels are chosen using test-set signal statistics, the encoding correlations in Fig. 2 and the comparisons in Table I are optimistically biased. The voxel selection must be performed on the training set only, or the entire pipeline (including voxel selection) must be cross-validated, to support the claimed generalizability.","section":"Section II-A (voxel selection)"},{"comment":"All results are on subject-averaged fMRI with a single train/validation/test split. The Mann-Whitney U tests in Table I compare distributions over voxels or frames within that one split; they do not provide evidence about stability across subjects or cross-validation folds. The paper's contribution (3) claims robustness based on the splitting method, but without reporting per-subject encoding performance or repeated cross-validation, the generalization claim is not substantiated. Please provide error bars across subjects (at least for the encoder correlation) or use leave-one-movie-out cross-validation for the central encoding and decoding metrics.","section":"Section II-A, Section III-A"}],"minor_comments":[{"comment":"The sentence 'We exclusively considered the visual and divided the films into N chunks of 32 RGB frames' is incomplete; 'visual' should be 'visual modality' or 'visual frames.'","section":"Section II-A"},{"comment":"The caption for Fig. 3 states 'SSIM scores between true and decoder-predicted movie frames' but does not mention that the decoder input is encoder-predicted fMRI, which is essential for interpreting the result.","section":"Fig. 3"},{"comment":"The entries 'Dssim' and 'Dmse' for the encoder-only models are given as '/', but no footnote explains that no decoder was trained; add a note.","section":"Table I"},{"comment":"The notation 'cos(̸ (v, ˆv))' is unusual and potentially a typo; please define whether it is cosine distance or cosine similarity and use standard notation such as '1 - cos_sim' if distance is meant.","section":"Eq. (1)"},{"comment":"The caption reports an 'average correlation score of 0.203' while Table I reports a median Ecorr of 0.202; clarify whether the stated value is the mean or the median and ensure consistency.","section":"Fig. 2 caption"},{"comment":"Kupershmidt et al. is cited as an arXiv preprint; please check whether a peer-reviewed version is available and cite that version if so.","section":"Reference [10]"}],"recommendation":"major_revision","confidential_remarks":"The reader's reject verdict is understandable given the missing ground-truth fMRI decoding experiment. I believe this is fixable within the manuscript's scope by adding that experiment and cross-validating the voxel selection; hence I recommend major revision rather than reject. I also note that the authors compare with [10] but do not provide code, which may be a reproducibility concern for a methods paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the fMRI movie decoding paper (2507.12009). Here's my take: the encoding part is a reasonable incremental contribution, but the paper's headline claim—reconstructing frames from neural activity—doesn't hold as written. The decoder never sees measured fMRI at train or test time. In the end-to-end setup (Eq. 3), the decoder's input is the encoder's predicted voxel values, which are a deterministic function of the movie frames. So the SSIM of 0.320 and the saliency maps characterize an autoencoder of the film frames through a voxel-shaped bottleneck, not a brain-reading pipeline. The abstract and discussion say 'from neural activity' and 'fMRI readout,' but no experiment feeds ground-truth fMRI (even subject-averaged) into the decoder. That's a load-bearing gap, easily fixable: run the trained decoder on actual test fMRI and report the SSIM against the null.\n\nWhat's genuinely new: they bring this temporal CNN encoder-decoder to the Emo-FilM dataset, a new naturalistic movie-fMRI dataset, and they add a saliency analysis that points to middle occipital, fusiform, and calcarine regions. The encoding analysis—held-out movie, correlation of 0.203 versus shuffled null—is plausible evidence that the encoder captures some shared visual response, though the SNR-based voxel selection is done on the full dataset before splitting, so test information leaks into feature selection. That's a mild circular step; a per-fold or training-only SNR would be cleaner. The Mann-Whitney tests on group-averaged voxel correlations also treat non-independent samples as independent, but that's secondary.\n\nIf I were an editor, I wouldn't desk-reject this. The encoder result and the dataset description are worth referee time, and the decoding flaw is addressable with one additional experiment. But as submitted, the central claim is overstated. I'd tell the authors to test the decoder on measured fMRI, reframe the reconstruction claim relative to that result, and redo the SNR selection without test data.\n\nFor your purposes: worth a read if you want a clean example of the 'predicted-fMRI decoder' pitfall in naturalistic decoding. I wouldn't cite it in a methods section until the ground-truth decoding test is added.","headline":"The decoding result is an autoencoder of movie frames through the encoder's predicted voxels, not a readout of measured brain activity; the encoding half is plausible but the central reconstruction claim is unsupported.","tokens_in":8288,"tokens_out":3026,"would_cite":false,"duration_ms":34264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an end-to-end CNN can predict visual-cortex fMRI responses to movie frames and reconstruct the watched frame from that predicted activity, including for a held-out film.","keywords":["fMRI","neural encoding","brain decoding","deep learning","movie watching","temporal convolutional networks","naturalistic stimuli","visual cortex"],"falsifier":"Feed the trained decoder the actual measured, subject-averaged fMRI volumes from the test set, bypassing the encoder, and compare the SSIM of the resulting reconstructions with the reported 0.320; if scores fall to near the shuffled baseline, the decoder was reading the encoder's latent code rather than neural activity.","tokens_in":7266,"feed_emoji":"🧠","tokens_out":8800,"duration_ms":88141,"temperature":0.7,"pith_summary":"The paper argues that a single end-to-end convolutional network can handle both directions of the brain–movie mapping: encoding film frames into predicted fMRI voxel activity, and decoding that activity back into a representative movie frame. Natural movies run at roughly 32 frames per 1.3-second fMRI volume, so the encoder uses temporal convolutions to bridge the resolution gap instead of collapsing each chunk into an average frame. The authors report that the encoder reaches a mean correlation of 0.203 with measured voxel activity on test volumes and that the decoder reaches a mean SSIM of 0.320 against the true middle frame, with generalization to a completely held-out movie. They also find that the decoder's saliency maps concentrate in the middle occipital area, fusiform area, and calcarine cortex, consistent with reconstructing shapes, faces, edges, and contrasts. If the claims are right, the model offers a way to use deep network behavior as a proxy for probing how the human visual system processes naturalistic films.","feed_headline":"Decoder rebuilds movie frames from predicted brain activity","feed_subtitle":"Temporal convolutions bridge 32 frames per scan; saliency maps point to face-, shape-, and edge-selective areas.","key_machinery":"The machinery is the temporal encoder-decoder architecture: the encoder stacks temporal convolutional layers over 32-frame chunks of a film and outputs a 4609-dimensional voxel vector, while the decoder is a CNN that maps that voxel vector to a single target frame, defined as the chunk's middle frame. The encoder loss is $\\mathcal{L}_E = \\mathrm{MSE}(\\mathbf{v}, \\hat{\\mathbf{v}}) + 0.5 \\cdot \\cos\\text{-distance}(\\mathbf{v}, \\hat{\\mathbf{v}})$, and the decoder loss is $\\mathcal{L}_D = 0.35\\,\\mathcal{L}_{psim} + 0.35\\,\\mathcal{L}_{ssim} + 0.30\\,\\mathcal{L}_{tv}$, with a combined loss $\\mathcal{L}_{ED} = 0.5\\,\\mathcal{L}_E + 0.5\\,\\mathcal{L}_D$. Temporal convolution is what lets the model bridge the mismatch between 32 frames per second of film and one fMRI volume every 1.3 seconds, and the decoder-as-regularizer is what ties reconstruction quality back to encoding quality. The saliency maps are produced by backpropagating SSIM gradients through the decoder and summing absolute saliencies across all test reconstructions.","core_discovery":"The central claim is that an end-to-end trained encoder-decoder CNN, with temporal convolutional layers in the encoder and a fully convolutional decoder, predicts subject-averaged fMRI activity in visual cortex from chunks of 32 consecutive movie frames and reconstructs the middle frame of each chunk from the encoder's predicted activity, including for a film completely excluded from training. The decoder is fed the encoder's predicted voxel values rather than measured fMRI, and its loss combines perceptual similarity, SSIM, and total variation; the combined encoder-decoder loss uses the decoder as a regularizer that improves encoder predictions over an encoder trained alone. Saliency analysis by backpropagating SSIM gradients locates the most influential voxels in the middle-superior occipital area, fusiform gyrus, and calcarine cortex, which the authors connect to shape, face, and basic-feature processing respectively. The paper presents this as evidence that a deep model can serve as a proxy for studying visual representation in naturalistic movie watching.","pith_inferences":["A direct test of the reconstruction claim would be to run the decoder on measured, subject-averaged test volumes rather than encoder-predicted activity, since the reported reconstructions never bypass the encoder.","Because all fMRI was averaged across 30 subjects, the reconstruction likely reflects shared and stereotyped responses rather than an individual viewer's brain activity; subject-level decoding is a natural next step that would require handling lower signal-to-noise ratios.","The saliency-region story could be tested directly by lesioning the top-saliency voxels in the predicted fMRI volumes and measuring how much SSIM degrades, which would connect causal perturbation to the reported correlational saliency maps.","The same temporal-convolution design could transfer to other naturalistic modalities or to datasets with different preprocessing, although the authors note that cross-dataset transfer is not guaranteed."],"forward_implications":["The same model can predict visual-cortex fMRI from movie frames and reconstruct a representative frame from that predicted activity, including on a held-out film never seen during training.","Temporal convolutions over 32-frame chunks offer a way to handle the frame-rate-to-TR resolution gap without averaging away the movie's fast visual dynamics.","Adding the decoder as a regularizer improves the encoder's voxel predictions compared with training the encoder alone, so reconstruction quality can be used to sharpen encoding.","Decoder saliency maps implicate shape-, face-, and edge-selective regions, suggesting the reconstructions are driven by the same broad visual features the human visual system uses.","A successful encoder-decoder pair of this kind provides a proxy through which deep network behavior can be used to probe visual processing in naturalistic stimuli."],"supporting_citations":[{"why":"Provides the Emo-FilM dataset of 30 subjects watching 14 short films, the data all experiments are trained and evaluated on.","marker":"[14]"},{"why":"Supplies the architecture inspiration, the decoder loss weights, and the self-supervised reconstruction approach this work extends.","marker":"[10]"},{"why":"Defines the perceptual similarity metric used in the decoder loss.","marker":"[4]"},{"why":"Provides the pretrained VGG16 network whose five layers are used to compute the perceptual similarity loss.","marker":"[5]"},{"why":"Supplies the Schaefer 1000 parcellation used to build the visual-cortex voxel mask.","marker":"[15]"},{"why":"Provides the AAL3 atlas used to label the anatomical regions in the saliency analysis.","marker":"[16]"},{"why":"Documents spurious reconstructions from brain activity, motivating the CNN-based approach as less prone to hallucination.","marker":"[12]"}],"fun_headline_variants":["Deep model rebuilds movie frames from predicted fMRI activity","Temporal convolution maps film frames to brain activity and back","Encoder-decoder CNN reconstructs movie visuals from brain signals","Saliency maps show face, shape, edge areas drive fMRI decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decoder is trained and evaluated on the model's own predicted brain signals, not on measured brain signals, so the reconstruction-from-brain-activity claim assumes those predicted signals faithfully stand in for real measurements.","fun_headline_variants_meta":{"raw":{"variants":["Deep model rebuilds movie frames from predicted fMRI activity","Temporal convolution maps film frames to brain activity and back","Encoder-decoder CNN reconstructs movie visuals from brain signals","Saliency maps show face, shape, edge areas drive fMRI decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1802,"prompt_tokens":940,"completion_tokens":862,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":794}},"tokens_in":556,"tokens_out":862,"duration_ms":9666,"temperature":1.0,"reasoning_tokens":794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:55:52.312949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the trained decoder the actual measured, subject-averaged fMRI volumes from the test set, bypassing the encoder, and compare the SSIM of the resulting reconstructions with the reported 0.320; if scores fall to near the shuffled baseline, the decoder was reading the encoder's latent code rather than neural activity.","supporting_citations":[{"cited_title":"Emo-film: A multimodal dataset for affective neuroscience using naturalistic stimuli,","cited_arxiv_id":null,"evidence_quote":"Provides the Emo-FilM dataset of 30 subjects watching 14 short films, the data all experiments are trained and evaluated on."},{"cited_title":"Self-supervised natural image reconstruction and large-scale semantic classification from brain activity,","cited_arxiv_id":null,"evidence_quote":"Defines the perceptual similarity metric used in the decoder loss."},{"cited_title":"Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri,","cited_arxiv_id":null,"evidence_quote":"Supplies the Schaefer 1000 parcellation used to build the visual-cortex voxel mask."},{"cited_title":"Automated anatomical labelling atlas 3,","cited_arxiv_id":null,"evidence_quote":"Provides the AAL3 atlas used to label the anatomical regions in the saliency analysis."},{"cited_title":"Spurious reconstruction from brain activity,","cited_arxiv_id":null,"evidence_quote":"Documents spurious reconstructions from brain activity, motivating the CNN-based approach as less prone to hallucination."}],"review_version":1}