{"id":"2d23b417-17b7-4894-ba20-2500a62a1575","arxiv_id":"2507.19052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On the Algonauts 2025 benchmark, a linear brain-encoding model with X-CLIP and Whisper features outperformed an attention-based model on out-of-distribution movies, and linguistic features were excluded after preliminary analyses.","lead":"This paper tests brain-encoding models that predict brain activity from movie clips, comparing complex attention-based models with simpler linear models on familiar and novel videos. It reports that the simpler model generalizes better to novel movies and that text features did not help predictions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD trade-off and linguistic-redundancy claims rest on small absolute correlation differences and an unreported preliminary analysis; no significance tests or error bars are provided.","rationale":"The reader's weakest-assumption analysis correctly identifies the two fragile pillars of the paper: the OOD superiority of the linear model and the redundancy of linguistic features. My stress-test pass finds the same two issues and no additional independent objection. The absolute OOD correlations are small, no significance testing is reported, and the linguistic-feature exclusion in Section 4.3 is described only qualitatively as a preliminary finding with no supporting numbers. These are empirical-evidence gaps rather than logical inconsistencies, so a conditional verdict remains appropriate: the claims are plausible and the experimental setup is reasonable, but they should not be treated as established until the missing statistical support is supplied. The concrete permutation test and the BERT-inclusion rerun would directly resolve whether the headline claims survive. Therefore, I recommend keeping the reader's CONDITIONAL verdict unchanged.","tokens_in":7800,"tokens_out":1905,"duration_ms":22641,"concrete_test":"Using the existing OOD predictions, compute the per-parcel Pearson correlation for the linear, attention, and baseline models for each subject, then run a paired permutation test across the 1,000 parcels (with subject-stratified resampling or per-subject blocking) comparing linear versus baseline and linear versus attention. Report bootstrapped 95% confidence intervals for the mean difference. If the 0.105 versus approximately 0.089 gap is not significant at a pre-registered threshold, the complexity-generalization trade-off is not established. Separately, rerun the final pipeline with PCA-reduced BERT features included and report ID and OOD correlations with the same uncertainty quantification; if the change is within noise, the linguistic-redundancy claim survives, and if it is not, the exclusion in Section 4.3 needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a complexity-generalization trade-off depends on the OOD result where the linear model reaches 0.105 average Pearson versus 0.098 for the attention model and roughly 0.089 for the baseline (an 18% relative gain). These absolute differences are small, yet the paper provides no per-parcel error bars, confidence intervals, or significance tests. Because the average is taken over 1,000 parcels, a paired permutation test or bootstrap interval could easily show that the linear model's edge is within noise. The paper's auxiliary claim that linguistic features are redundant is even less supported: Section 4.3 states that preliminary encoding models with BERT features 'did not yield a significant improvement' and were therefore excluded, but no numbers, error bars, or statistical test for that comparison are reported. That claim is load-bearing for the title and abstract, which assert the dominance of audiovisual streams over textual information, yet it is unverifiable from the manuscript as written. The ID result (0.218 vs 0.212 vs 0.203) similarly lacks any indication that the ordering is stable, and the 18% figure is a relative improvement over a very small absolute difference, which is easy to overstate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents multimodal brain encoding models for fMRI responses to naturalistic movies, using X-CLIP visual features, Whisper auditory features, and BERT linguistic features, combined through either a linear regression or an attention-based fusion network. The models are evaluated on the Algonauts 2025 challenge data, with in-distribution (ID) testing on Friends season 7 and out-of-distribution (OOD) testing on six diverse movies and documentaries. The central claims are that (i) a higher-capacity attention model performs best on ID data while a simpler linear model generalizes better on OOD data, outperforming a baseline by 18%; (ii) linguistic features do not improve predictive accuracy; and (iii) Whisper-based auditory features provide marked gains in auditory cortex. The paper argues for a model-complexity versus generalization trade-off in brain encoding and for the primacy of audiovisual streams over textual information in familiar-language naturalistic perception.","tokens_in":8086,"tokens_out":3353,"duration_ms":32888,"significance":"If the reported findings are statistically robust, the paper makes a useful contribution by demonstrating the value of out-of-distribution evaluation in brain encoding and by comparing linear versus attention-based fusion mechanisms in a large-scale naturalistic setting. The use of domain-specific pretrained feature extractors (X-CLIP, Whisper) and the spatial analysis of auditory cortex improvements are strengths, as is the explicit testing on a designated OOD set. The central trade-off claim and the linguistic-redundancy claim are interesting and would likely influence future encoding-model design. However, the support for these claims is currently weakened by the absence of error bars and significance tests on the central comparisons, and by the reliance on unreported preliminary analyses for key design decisions, including the exclusion of linguistic features. The potential significance is therefore real but conditional on improved statistical reporting.","major_comments":[{"comment":"The central OOD claim rests on average Pearson correlations of 0.105 (linear model), 0.098 (attention-based model), and approximately 0.089 (baseline) across 1000 parcels. These absolute differences are small, yet no error bars, confidence intervals, or significance tests are reported. Because the average is over many parcels, a paired permutation test or bootstrap interval across parcels and/or subjects could reveal whether the linear model's edge is within noise. Without such an analysis, the asserted 18% improvement and the associated complexity-generalization trade-off are not statistically supported.","section":"Section 5.3, Figure 5, Table 1"},{"comment":"The exclusion of linguistic features is justified by 'preliminary encoding models' in which BERT features 'did not yield a significant improvement,' but no quantitative results, error bars, or test statistics are provided for this comparison. This claim is load-bearing for the abstract and title, which assert the dominance of audiovisual streams over textual information. The authors should either report the full linguistic-feature comparison (with numbers and significance tests) or explicitly moderate the claim to reflect that the evidence is preliminary and not independently verifiable from the manuscript.","section":"Section 4.3"},{"comment":"Several design decisions that plausibly affect the main results are justified only by 'preliminary analyses' with no details: the selection of the 23rd X-CLIP transformer block (Section 4.1), the number of previous time points Nd=10 (Section 4.4), and the number of PCA components per modality (Section 4.4). Without reporting the hyperparameter search ranges, the validation metric used, and the selected values, the reproducibility of the pipeline is incomplete, and readers cannot assess how sensitive the OOD trade-off and the linguistic-feature conclusion are to these choices.","section":"Sections 4.1 and 4.4"}],"minor_comments":[{"comment":"The heading 'Dateset and Challenge' contains a typo; it should read 'Dataset and Challenge.'","section":"Section 3, heading"},{"comment":"The caption contains an incomplete word: 'the baselin' appears instead of 'the baseline.'","section":"Figure 3 caption"},{"comment":"The expression 'seasons 1˘− 6' appears to be a formatting artifact; it should read 'seasons 1–6.'","section":"Section 3, first paragraph"},{"comment":"The cost function omits the bias term b that appears in Equation (2); either include it in the squared-error expression or clarify that the bias is absorbed into the weight vector.","section":"Equation (3)"},{"comment":"The glass-brain visualizations use different color-bar ranges between the top and bottom panels, which makes direct visual comparison of model performance misleading; the color bars should be normalized to a common range or the differences should be explicitly stated.","section":"Figures 4 and 6"},{"comment":"Table 1 reports OOD performance only for the proposed linear model, not for the attention-based model or the baseline, limiting the reader's ability to assess stimulus-specific differences across models; consider adding columns for all models.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is currently a typical conference-style paper with a strong empirical setup but insufficient statistical rigor for its central claims. The repeated reliance on 'preliminary analyses' for the linguistic-feature exclusion and hyperparameter choices is a reproducibility concern that should be addressed before publication. The authors should be encouraged to add paired significance tests or bootstrap intervals for the ID and OOD comparisons, and to report the linguistic-feature ablation with numbers. I do not see evidence of circularity, as the predictions are on held-out data and the features come from externally pretrained models; the main issue is the lack of uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper asks a good question—whether a simpler fusion model generalizes better than a complex attention model on out-of-distribution movies—and it builds a clean pipeline with X-CLIP and Whisper features on Algonauts 2025. That part is worth engaging with. The problem is that the headline claims, especially the 18% OOD improvement and the redundancy of linguistic features, are not backed by error bars or significance tests, and one of them rests on an analysis that is described but never shown.\n\nThe genuine strengths: the OOD test is real held-out data, not a synthetic shift. The features come from externally pretrained models, so no circularity. The spatial finding—Whisper features help most in auditory cortex—is plausible and consistent with prior work. The paper is also honest in Section 4.3 that BERT features were tried and excluded after a preliminary analysis; that is more transparent than most, but it still hides the actual numbers.\n\nNow the soft spots, in proportion. The central trade-off claim depends on 0.105 vs 0.098 vs 0.089 average Pearson. Those are small absolute differences, and with 1,000 parcels, a paired permutation test or bootstrap could easily show the linear model's edge is within noise. The paper says the linear model was better for all four subjects, which is suggestive, but without variance, it is not convincing. The linguistic-redundancy claim is even weaker: we only get a sentence saying BERT features 'did not yield a significant improvement.' No correlations, no p-values, no subject-level numbers. That is load-bearing because the title and abstract claim audiovisual dominance over text. You cannot verify that from the manuscript. Also, several hyperparameters (X-CLIP block index, PCA components, Nd) were chosen from unreported 'preliminary analyses,' which makes the whole pipeline harder to reproduce.\n\nI would take the trade-off as a hypothesis, not a demonstrated result. But the setup is sound and the question matters for neuro-AI model selection. The paper deserves a serious referee, though it needs heavy revision: add per-parcel error bars and paired tests for the model comparisons, and report the BERT inclusion numbers with the same care as the main results. If the findings survive that, they will be a solid contribution. As it stands, I would not cite the current version for the strong claims, but I would cite it for the OOD benchmark methodology if it is fixed. Bring it to reading group as an example of why null results need the same statistical rigor as positive ones.","headline":"A useful OOD comparison of fusion architectures for brain encoding, but the headline claims rest on small correlation differences and an unreported BERT analysis.","tokens_in":8592,"tokens_out":1478,"would_cite":false,"duration_ms":17349,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Predicting brain activity from movies: a simple linear fusion of visual and auditory features generalizes better to out-of-distribution films than an attention-based network, while language features add no measurable predictive gain.","keywords":["brain encoding","multimodal fMRI","out-of-distribution generalization","visual-auditory fusion","naturalistic stimuli","linear versus attention models","Whisper speech features","X-CLIP video features"],"falsifier":"Retrain both models with and without BERT features on repeated random train/OOD splits; if the BERT-augmented model ever matches or beats the audiovisual-only model on a held-out set, or if the linear model's OOD edge over the attention model does not replicate across subjects and seeds, the paper's central claims are refuted.","tokens_in":7684,"feed_emoji":"🧠","tokens_out":6879,"duration_ms":66711,"temperature":0.7,"pith_summary":"This paper asks whether brain-encoding models trained on naturalistic movies can predict fMRI responses to films they have never encountered, and what stimulus features carry that prediction. Using pretrained visual (X-CLIP) and auditory (Whisper) features, it compares a simple linear fusion model against a high-capacity attention-based fusion network on both in-distribution and out-of-distribution movies. The central finding is a reversal: the attention model wins on the familiar sitcom test set (0.218 average Pearson), while the linear model generalizes better to six diverse unseen films (0.105, an 18% gain over the baseline), suggesting a complexity-generalization trade-off. The paper also reports that BERT language features added no predictive improvement, so the final models use only the audiovisual streams, pointing to a dominance of continuous sight and sound over text in familiar-language perception. The reason to care: if true, robust neuro-AI encoding should be benchmarked out of distribution, and language encoders may be redundant in audiovisual-rich settings.","feed_headline":"A simple linear model predicts brain activity best on unseen movies","feed_subtitle":"Across six unseen films, the audiovisual linear encoder beats the benchmark baseline by 18 percent.","key_machinery":"The load-bearing construction is the parcel-wise encoding pipeline: each fMRI time point is predicted from the previous 10 time points' features to account for the hemodynamic delay, with X-CLIP video features (1024-dimensional after spatial and temporal pooling) and Whisper audio features (per-segment averaged encoder states) reduced by PCA and concatenated. Two fusion heads are compared: a plain linear regression per parcel and a three-stage attention network (modality-specific multi-head self-attention, a Squeeze-and-Excitation-style feature-wise gate, and a three-layer MLP prediction head). The decisive experimental mechanism is the split between the in-distribution sitcom test set and the out-of-distribution set of six stylistically different films, which produces the claimed performance reversal.","core_discovery":"The paper's central claim is that a simple linear regression over delayed, PCA-reduced visual and auditory features is the more generalizable brain encoder: on the out-of-distribution set it reaches 0.105 mean Pearson correlation, beating the benchmark baseline by 18%, while the attention-based fusion model, though best in-distribution (0.218 vs 0.212 for linear and 0.203 for baseline), falls to 0.098 OOD and is outperformed by the linear model in all four subjects. The paper further claims that including BERT-subtitle features did not improve predictive accuracy in preliminary tests and increased overfitting risk, so linguistic content was excluded; the observed gains were spatially concentrated in auditory cortex, attributed to Whisper's speech representations. The paper frames these results as evidence for a complexity-generalization trade-off, for the redundancy of linguistic streams in familiar-language audiovisual perception, and for the necessity of OOD testing in encoding-model evaluation.","pith_inferences":["If the audiovisual-dominance result is real, language features should matter more in unfamiliar-language movies or in dialogue-free clips, a split the paper does not test; that contrast would sharpen the boundary of the claim.","The linear model's OOD advantage could come from PCA regularization rather than linearity per se; ablating PCA dimensionality across both models would separate those causes.","The same comparison could be run on musical or narration-light documentaries to test whether Whisper's speech focus, rather than acoustic fidelity, drives the auditory-cortex gains.","With only four subjects, the reversal may be subject-specific; a per-subject random-effects analysis would tell whether the trade-off is consistent at the population level."],"forward_implications":["Neuro-AI benchmarks should report out-of-distribution performance alongside in-distribution accuracy, since in-distribution winners can be the weakest OOD models.","Audiovisual encoding models for familiar-language content can drop language encoders entirely, lowering complexity and overfitting risk without sacrificing predictive power.","Pretrained speech encoders such as Whisper are worth the cost: they produce the largest gains in auditory cortex relative to the baseline.","Architectural capacity is not the main driver of robust encoding; a fixed linear map over good features can beat a nonlinear fusion network on novel stimuli.","Stimulus congruence matters: the linear model tracks auditory cortex best when narration and visuals align, and worst for silent black-and-white film."],"supporting_citations":[{"why":"Defines the benchmark task, the train/test movie split, and the linear baseline this paper compares against.","marker":"[10]"},{"why":"Supplies the large-scale fMRI movie-watching dataset used for training and evaluation.","marker":"[11]"},{"why":"Provides the 1,000-region cortical parcellation that defines the prediction targets.","marker":"[12]"},{"why":"Supplies the X-CLIP visual feature extractor used as the visual stream.","marker":"[13]"},{"why":"Supplies the Whisper auditory feature extractor credited with the auditory-cortex gains.","marker":"[15]"},{"why":"Provides the BERT linguistic features whose exclusion is a central result.","marker":"[16]"},{"why":"Defines the SlowFast visual features used by the baseline model.","marker":"[18]"},{"why":"Describes the Squeeze-and-Excitation gating mechanism on which the feature-wise attention module is based.","marker":"[17]"}],"fun_headline_variants":["Simple beats complex: Linear model wins on unseen movie data","Audiovisual linear encoder trumps complex model on unseen films","Less is more: Simple encoder generalizes better to new movies","Why simple audio-visual models beat complex ones on unseen data","Linear beats deep fusion on six novel movies (18% gain)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the preliminary, unreported comparison of BERT features actually showed no predictive gain, and that the small OOD correlation differences (0.105 vs 0.098 vs about 0.089 baseline) are real rather than noise, since no significance tests are given.","fun_headline_variants_meta":{"raw":{"variants":["Simple beats complex: Linear model wins on unseen movie data","Audiovisual linear encoder trumps complex model on unseen films","Less is more: Simple encoder generalizes better to new movies","Why simple audio-visual models beat complex ones on unseen data","Linear beats deep fusion on six novel movies (18% gain)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000719,"raw_usage":{"total_tokens":3242,"prompt_tokens":974,"completion_tokens":2268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2183}},"tokens_in":590,"tokens_out":2268,"duration_ms":16950,"temperature":1.0,"reasoning_tokens":2183,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:00:56.398982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain both models with and without BERT features on repeated random train/OOD splits; if the BERT-augmented model ever matches or beats the audiovisual-only model on a held-out set, or if the linear model's OOD edge over the attention model does not replicate across subjects and seeds, the paper's central claims are refuted.","supporting_citations":[{"cited_title":"Slowfast networks for video recognition","cited_arxiv_id":null,"evidence_quote":"Defines the SlowFast visual features used by the baseline model."}],"review_version":2}