Pith. sign in

REVIEW 4 major objections 8 minor 24 references

Multi-modal brain encoding models for multi-modal stimuli

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-modal AI embeddings beat unimodal video at predicting movie-watching brains.

desk verdict A solid empirical benchmark undercut by an over-interpreted linear residual analysis; the multi-modal-vs-unimodal comparison is worth refereeing, but the modality-attribution claims need a nonlinear check or softer wording. read the letter →

arxiv 2505.20027 v1 pith:FIWVFDTR submitted 2025-05-26 q-bio.NC cs.AIcs.CLcs.LGeess.ASeess.IV

Subba Reddy Oota , Khushbu Pahwa , Mounika Marreddy , Maneesh Singh , Manish Gupta , Bapi S. Raju This is my paper
classification q-bio.NCcs.AIcs.CLcs.LGeess.ASeess.IV
keywords brainencodingfMRImulti-modalmodelscross-modalpretrainingjointlypretrainedresidualanalysisnaturalisticmoviestimulialignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether multi-modal AI models track the brain better than single-modality models when a viewer watches a real movie with both picture and sound. Using fMRI from the Movie10 dataset, it compares three video-only models, two speech-only models, one cross-modal model (ImageBind), and one jointly pretrained model (TVLT), all mapped to brain responses with ridge regression. The central result is that both types of multi-modal embeddings align significantly better than unimodal models in language regions (AG, PCC, PTL, IFG) and visual regions (EVC, MT), and residual analysis shows the cross-modal advantage comes mainly from video information while the joint advantage comes from both video and audio. This matters because it points to which brain regions integrate across senses and gives a concrete benchmark for choosing model representations in naturalistic neuroimaging.

What carries the argument

Voxel-wise ridge regression encoding models map each stimulus representation to fMRI responses per subject, with normalized alignment scores capped by cross-subject prediction accuracy. The distinctive mechanism is residual analysis: a linear regressor learns the video-only or speech-only contribution inside the multi-modal embedding, subtracts it, and the leftover representation is scored against the brain again. A drop in alignment after removal tells which modality carried the brain-relevant signal, and a remaining alignment shows information beyond the unimodal features under test.

What would settle it

Train a video-only model with the same encoder architecture, parameter count, and pretraining data as ImageBind's video encoder and compare its brain alignment with ImageBind's concatenated embedding; if the unimodal model matches or exceeds it, the claimed multi-modal advantage would collapse, and a nonlinear residual regressor that destroys the leftover alignment would further show the 'additional information' is an artifact of linear subtraction.

Watch

Extended reading notes

Core claim

The paper claims that when people watch movies with sound, representations from multi-modal Transformers predict their fMRI responses better than representations from video-only or speech-only models, in a specific set of language and visual regions. Using ImageBind as a cross-modal model and TVLT as a jointly pretrained model, the authors find that the extra alignment does not simply rest on one modality: removing video features from the cross-modal model's concatenated embedding drops alignment in AG and MT, while removing audio features does not; for the jointly pretrained model, removing either modality partially reduces alignment. The paper interprets this as evidence that multi-modal models capture brain-relevant information beyond any single unimodal embedding, and that the two training strategies differ in how they weight the modalities.

Load-bearing premise

The comparison assumes the chosen unimodal video and speech models are strong, representative baselines; the paper itself notes in Appendix O that architectural and pretraining variability mean future tightly controlled comparisons could shrink the multi-modal advantage.

Editorial extensions

If this is right

  • Encoding models for naturalistic viewing should use multi-modal representations, since they capture brain-relevant variance that unimodal video or speech models miss in language and visual regions.
  • The angular gyrus appears to act as a multi-modal buffer: its alignment with the joint model survives removal of either modality, whereas its alignment with the cross-modal model depends on video features.
  • Cross-modal pretraining strategies bias the model toward the visually richest modality, so conclusions about multi-modal brain alignment must be checked per modality.
  • Unimodal speech models provide weaker brain-relevant semantics than video or multi-modal models in almost every tested region, with object visual cortex being the exception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear-removal story holds, a nonlinear control should be run; a richer probe might find audio information in the cross-modal residual that linear regression cannot see, changing the attribution from video-only to both modalities.
  • The paper's logic predicts that fine-tuning a jointly pretrained model on a brain-alignment objective should further increase alignment in AG, PCC, PTL, and IFG while keeping its balance across modalities; this is testable with the same Movie10 data.
  • Matched-architecture ablations are the natural next step: swapping ImageBind's video encoder for an equally large video-only backbone with identical pretraining data would isolate whether the multi-modal advantage comes from joint training or from a stronger visual backbone.
  • The AG-buffer claim suggests a clinical extension: patients who acquire vision or hearing late in life should show brittle AG activation for the late-acquired modality, mirroring the cross-modal model's behavior; the paper hints at this but does not test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper develops brain encoding models for fMRI data from participants watching movies with audio, comparing unimodal video (ViT-B, VideoMAE, ViViT) and speech (Wav2Vec2, AST) representations against multi-modal representations from a cross-modal model (ImageBind, with video and audio embeddings concatenated) and a jointly pretrained model (TVLT). Using ridge regression voxel-wise encoding with a clean train/test split across movies and noise-ceiling normalization, the authors find that multi-modal models improve brain alignment in several language and visual regions relative to unimodal models. They then perform residual analyses, removing unimodal features from multi-modal representations, and conclude that cross-modal alignment is partially attributable to video features, while jointly pretrained alignment is partially attributable to both video and audio features, and that there is additional information beyond the unimodal embeddings.

Significance. If the conclusions hold, the paper would be a useful contribution to the use of multi-modal pretrained models as proxies for human multi-modal processing, and the Movie10 dataset is a valuable testbed. The strengths are: clean cross-movie train/test split, noise-ceiling normalization, block permutation tests, and publicly released code. However, the interpretative claims about modality attribution are not yet supported by the analyses as presented due to the linear-only removal and uncontrolled model comparisons.

major comments (4)
  1. [Sec. 5 and Sec. 6.3 (Fig. 4)] The central claim that multi-modal representations contain 'additional information beyond the unimodal embeddings' (Abstract; Sec. 6.3) rests on residual analysis in which only the linear projection of unimodal features onto the multi-modal representation is removed (Sec. 5, Fig. 1B). Because the subsequent brain encoder is also linear (ridge regression), a nonlinear encoding of the same modality content within the multi-modal representation can survive this removal and still drive the linear brain alignment. The quality check reported in Sec. 6.3 (Pearson correlation as low as 0.56 between unimodal video features and the IB Concat feature space) does not establish that the unimodal information was removed; it indicates only a moderate linear relationship, and no check of the residual's dependence on the removed features (e.g., via correlation of the residual with the unimodal features, or via nonlinear residualization) is provided. Consequently, the modality-attribution conclusions (cross-modal alignment 'partially attributed to video'; TVLT 'partially attributed to both video and audio') are not established. I recommend adding a nonlinear residualization analysis (e.g., a nonlinear regressor with cross-validation) and/or reporting the residual's correlation with the removed features, or moving the variance-partitioning analysis of Appendix L into the main text and re-framing the conclusions accordingly.
  2. [Sec. 6.1 and Appendix O] The primary comparison of multi-modal versus unimodal models is confounded by architecture, parameter count, feature dimensionality, and pretraining data. For instance, IB Concat is 2048-dimensional (1024 video + 1024 audio) compared with 768-dimensional unimodal video/speech features; ImageBind's video encoder is a ViT-B initialized from the same architecture as the unimodal ViT-B but trained contrastively on different data; TVLT (88M parameters, pretrained on HowTo100M) differs from the unimodal models in both architecture and pretraining. Appendix O acknowledges that 'future work could benefit from more tightly controlled comparisons,' but the main-text statements (e.g., 'cross-modal embeddings contain additional information beyond the two modalities' in Sec. 6.1) are asserted without this caveat. A controlled comparison (e.g., matching backbone, dimensionality, and pretraining data across unimodal and multi-modal conditions) is needed to support the claim that multi-modal training per se improves brain alignment; otherwise the conclusions should be restricted to the specific models tested.
  3. [Sec. 6.3, Fig. 4] The 'partial attribution' conclusions require a statistical comparison between the full multi-modal alignment and the alignment after feature removal, but no such test is reported. The methods only describe comparing residual alignment to chance (Sec. 5). If the drop after removing, say, video features is not significant across participants (N=6), then the claim that the alignment is 'partially attributed to the video modality' is unsupported. The authors should report paired tests (e.g., Wilcoxon signed-rank) on the differences between full and residual alignment for each ROI, with multiple-comparison correction.
  4. [Sec. 6.3 and Fig. 1B] The residual analysis removes features from external unimodal models (e.g., ViT-B for video) rather than the modality-specific branches internal to the multi-modal model. For TVLT, removing 'Unimodal VM' features from the joint representation does not isolate the video stream as it is encoded within TVLT, since the joint encoder fuses video and audio tokens. The paper does include internal removals (e.g., 'IB Concat - IB Video', 'TVLT Joint - TVLT Video'), but the main-text narrative often refers to the external removals (e.g., 'removal of unimodal video embeddings from the IB Concat' in the Qualitative analysis). The interpretation of modality contributions should be based on the internal removals, with the external removals used only for the 'beyond unimodal embeddings' question.
minor comments (8)
  1. [General] The manuscript contains duplicated text in Sec. 6.3 (the section appears twice with slightly different wording) and figure captions still say 'Under review as a conference paper at ICLR 2025' despite the paper being published; please clean up the camera-ready version.
  2. [Sec. 6.3] The sentence 'We computed Pearson correlation where unimodal video features are projected onto the multi-modal IB Concat feature space using the residual approach' is ambiguous; clarify whether the reported 0.56 is the correlation between the original and projected CM features, or between the residual and the removed VM features, and state the interpretation of this value as a removal-quality check.
  3. [Appendix E] Appendix E refers to 'bootstrap ridge-regression (Appendix I)', but Appendix I discusses only the choice of ridge regression and does not describe the bootstrap procedure; specify how bootstrapping is applied (e.g., resampling TRs, voxels, or the regularization parameter).
  4. [Sec. 5] The z-score threshold for outlier removal is a free parameter but its value is not reported; provide the threshold used for both stimulus features and brain responses.
  5. [References and Sec. 3] The paper cites St-Laurent et al. (2023) for the 'Movie10' dataset, but the cited paper describes the cneuromod-THINGS dataset; cite the correct dataset paper for Movie10.
  6. [Fig. 2] In Fig. 2, the left-panel legend entries such as 'Random Unimodal VM' and 'IB Concat Unimodal VM Random' are confusing and the figure resolution is low; please provide a clearer legend and higher-resolution panels.
  7. [Abstract and Sec. 6.1] The abstract's phrase 'identifying which brain regions process unimodal versus multi-modal information' is a strong interpretation of encoding-model predictions; consider rephrasing to 'which brain regions are better predicted by unimodal versus multi-modal representations.'
  8. [Sec. 6.1 and Sec. 6.3] The paper states in Sec. 6.1 that 'when we average across visual regions, there is no additional information beyond unimodal video features,' but the residual analysis in Sec. 6.3 later suggests additional information in some visual regions (e.g., MT, EVC); reconcile these statements or clarify that they refer to different analyses.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the modality-attribution claims rest on held-out residual prediction with external pretrained models and a public fMRI dataset, not on fitted targets or self-citation chains.

full rationale

The paper's central claims are empirical comparisons between brain-alignment scores obtained from unimodal, cross-modal, and jointly pretrained model embeddings, evaluated on the public Movie10 fMRI dataset. The residual analysis in Sec. 5 and Sec. 6.3 fits a ridge regression r from unimodal features to multi-modal features on the training movies and then tests a separate encoder g' on held-out Life movie data; the residual predictions are not constructed to equal the original alignment scores. Thus the 'additional information beyond the unimodal embeddings' claim is a held-out empirical finding rather than a definitional equivalence. The residual analysis is linear-only, and the statement in Sec. 6.3 that a correlation of 0.56 means unimodal features were 'successfully removed' is statistically questionable because substantial linear dependence can remain; however, this is a validity and interpretation concern, not a circular reduction of the conclusion to its input. Appendix O openly acknowledges the limited control over architecture and pretraining differences, which further supports treating the main comparisons as empirical rather than forced by construction. The many self-citations (Oota et al., 2022b; Oota et al., 2023a; Oota & Toneva, 2023; Oota et al., 2024a) are method citations for the residual approach, which is also grounded in independently published work by Toneva et al. (2022) and Dong & Toneva (2023b), and they are not load-bearing uniqueness claims. No circularity pattern from the enumerated kinds is present; at most, the paper exhibits minor non-load-bearing self-citation and an unvalidated residual-removal criterion, which is a correctness risk rather than circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are proposed. The analysis rests on the validity of linear encoding models, the residual method, and the representativeness of the chosen pretrained models as baselines.

free parameters (3)
  • Ridge regularization lambda (L2 decay) = 10^1 to 10^3, selected on a 10% validation split
    Appendix E reports lambda tuning on a validation subset; a standard hyperparameter, not central to the conclusions.
  • Z-score threshold for outlier removal = not reported
    Section 5 states z-score thresholding is applied to stimulus and brain data, but the threshold value is not given, so the results depend on an unspecified choice.
  • Voxel inclusion threshold for normalized alignment = cross-subject prediction accuracy >= 0.05
    Section 5 selects voxels whose cross-subject accuracy is at least 0.05 before averaging normalized alignment; an arbitrary cutoff that affects aggregate scores.
assumptions (4)
  • domain assumption Linear ridge regression is an adequate encoding model for comparing stimulus representations to fMRI responses
    Section 5 uses voxel-wise ridge regression for all models; this is standard in the field, but nonlinear interactions between features and brain responses are not captured.
  • domain assumption Residual analysis via a linear map from unimodal features to multi-modal representations removes the modality's contribution to brain alignment
    Section 5 and Figure 1B regress unimodal features out of multi-modal representations; only linear contributions are removed, so nonlinear unimodal information can remain in the residuals.
  • domain assumption Cross-subject prediction accuracy provides a valid noise ceiling for normalization
    Section 3 adapts Schrimpf et al.'s noise ceiling method to Movie10; the ceiling is estimated from a finite number of subjects and assumes a perfect model, which may over- or under-estimate true ceiling.
  • domain assumption The selected unimodal and multi-modal models are representative comparators for their training paradigms
    Appendix O acknowledges differences in architecture and pretraining methods, yet the main-text comparison treats the chosen models as representative of unimodal versus multi-modal training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-modal brain encoding models for multi-modal stimuli." pith.science (2026). https://pith.science/paper/FIWVFDTR

@misc{pith2026250520027,
  author       = {Pith},
  title        = {Pith review of: Multi-modal brain encoding models for multi-modal stimuli},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIWVFDTR}},
  note         = {Machine review of arXiv:2505.20027}
}
read the original abstract

Despite participants engaging in unimodal stimuli, such as watching images or silent videos, recent work has demonstrated that multi-modal Transformer models can predict visual brain activity impressively well, even with incongruent modality representations. This raises the question of how accurately these multi-modal models can predict brain activity when participants are engaged in multi-modal stimuli. As these models grow increasingly popular, their use in studying neural activity provides insights into how our brains respond to such multi-modal naturalistic stimuli, i.e., where it separates and integrates information across modalities through a hierarchy of early sensory regions to higher cognition. We investigate this question by using multiple unimodal and two types of multi-modal models-cross-modal and jointly pretrained-to determine which type of model is more relevant to fMRI brain activity when participants are engaged in watching movies. We observe that both types of multi-modal models show improved alignment in several language and visual regions. This study also helps in identifying which brain regions process unimodal versus multi-modal information. We further investigate the contribution of each modality to multi-modal alignment by carefully removing unimodal features one by one from multi-modal representations, and find that there is additional information beyond the unimodal embeddings that is processed in the visual and language regions. Based on this investigation, we find that while for cross-modal models, their brain alignment is partially attributed to the video modality; for jointly pretrained models, it is partially attributed to both the video and audio modalities. This serves as a strong motivation for the neuroscience community to investigate the interpretability of these models for deepening our understanding of multi-modal information processing in brain.

Figures

Figures reproduced from arXiv: 2505.20027 by the authors.

Figure 1
Figure 1. (A) Overview of our proposed Multi-modal Brain Encoding Pipeline. Using fMRI recordings from participants watching popular movies included with speech, we align stimulus representations with brain recordings through ridge regression. For uni-modal alignment, we use representations from video models (VM) or speech models (SM), where the input consists exclusively of either videos (without speech) or speech, respectiv… view at source ↗
Figure 2
Figure 2. The Left plot compares the average normalized brain alignment of pretrained and randomly initialized multi-modal and unimodal models across whole brain. × indicates cases where pretrained model embeddings are significantly better than randomly initialized models, i.e., p≤ 0.05. Right plot displays the average normalized brain alignment for both multi-modal and unimodal model features specifically within language and… view at source ↗
Figure 3
Figure 3. Additionally, we cover visual regions like early visual cortex (EVC), scene visual areas (PPA) gg, y features. However, when compared to unimodal speech features, both multi-modal embeddings sho [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (13 more)
Figure 3
Figure 3. Figure 3: Average normalized brain alignment for video and audio modalities from multi-modal and ure 3: Average normalized brain alignment for video and audio modalities from multi-modal a iidl dlitfthlbid l ROIf l(AGPTL d IFG [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: Residual analysis: Average normalized brain alignment was computed across participants Language: AG 0.1 N Visual: MT 0.1 N Figure 4: Residual analysis: Average normalized brain alignment was computed acro [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Percent decrease of brain alignment after removal of unimodal embeddings from different [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Cross-subject prediction accuracy: (top) across whole brain, (bottom) across language, [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Average normalized brain alignment for per video and audio modalities from multi-modal [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Average normalized brain alignment for video and audio modalities from multi-modal [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Residual analysis for ATL, PTL, IFG, MFG, IFGOrb, PCC and dmPFC regions: Average [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Residual analysis for EVC, LOC, PPA, OFA and AC regions: Average normalized brain [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Normalized brain alignment across layers for multi-modal model (TVLT joint embeddings) [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: The plot compares the average normalized brain alignment across the whole brain under [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Whole Brain Analysis: Shared and Unique Variance explained between Cross-modal [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Language Network: Shared and Unique Variance explained between Cross-modal (IB [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Visual Network: Shared and Unique Variance explained between Cross-modal (IB Concat) [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

  1. [3]

    In each plot, Pink Area (Left Circle - Intersection) represents the unique variance explained by the multi-modal model that is not shared with the unimodal model

    in high-level visual regions (PPA, MT), they seem to 23 Published as a conference paper at ICLR 2025 0.323 0.2040.258 IB-Concat Unimodal VM 0.363 0.180.192 IB-Concat Unimodal SM 0.492 0.3190.074 TVLT Unimodal VM 0.504 0.280.089 TVLT Unimodal SM Figure 13: Whole Brain Analysis: Shared and Unique Variance explained between Cross-modal (IBConcat) and Unimoda...

    work page 2021

  2. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),

    work page 2019

  3. [6]

    Semantic scene descriptions as an objective of human vision

    Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209.11737,

    arXiv

  4. [8]

    Among unimodal speech models, the AST model shows better normalized brain alignment than the Wav2vec2.0 model

    Similar to the average results of unimodal video and speech models, we observe that multi-modal models exhibit better normalized brain alignment than individual unimodal video and speech models across language and visual regions. Among unimodal speech models, the AST model shows better normalized brain alignment than the Wav2vec2.0 model. Among unimodal v...

    work page 2025

  5. [11]

    The brain tells a story: Unveiling distinct representations of semantic content in speech, objects, and stories in the human brain with large language models

    Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Yamaguchi, Rieko Kubo, Shinji Nishimoto, and Yu Takagi. The brain tells a story: Unveiling distinct representations of semantic content in speech, objects, and stories in the human brain with large language models. bioRxiv, pp. 2024–02,

    2024

  6. [13]

    Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3220–3237, 2022a...

    work page 2022

  7. [14]

    Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp

    13 Published as a conference paper at ICLR 2025 Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007,

    work page 2025

  8. [15]

    Lxmert: Learning cross-modality encoder representations from transform- ers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transform- ers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5100–5111,

    work page 2019

Show all 24 references
  1. [17]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 543–553,

    2023

  2. [18]

    15 Published as a conference paper at ICLR 2025 A O VERVIEW OF APPENDIX SECTIONS • Section B: Cross-subject prediction accuracy • Section C: Detailed sub-ROIs of language, visual and auditory regions • Section D: Details of pretrained Transformer models • Section E: Implementa...

    2025

  3. [19]

    D D ETAILS OF PRETRAINED TRANSFORMER MODELS Details of each pretrained Transformer model are reported in Table 1 in Appendix

    C D ETAILED SUB -ROI S OF LANGUAGE , VISUAL AND AUDITORY REGIONS The data covers seven brain regions of interest (ROIs) in the human brain with the following sub- divisions: (i) early visual (EV: V1, V2, V3, V3B, and V4); (ii) object-related areas (LO1 and LO2); (iii) face-rel...

    2018

  4. [22]

    IB Concat Shuffle

    Observation from Fig. 11 indicates a consistent drop in performance from early to lower layers, specifically for both TVLT joint and unimodal video models. The key finding 19 Published as a conference paper at ICLR 2025 Language: ATL0.1 0.2 0.3 0.4 0.5 0.6 IB Concat TVLT Joint...

    2014

  5. [24]

    to those examining 101 models (Antonello et al., 2021), demonstrates that this approach is both valid and valuable for understanding the relationship between artificial and biological language processing. O L IMITATIONS The low alignment scores clearly show that despite the in...

    2021

  6. [103]

    Best λ was chosen by tuning on validation data that comprised a randomly chosen 10% subset from train set used only for hyper-parameter tuning. 17 Published as a conference paper at ICLR 2025 *^ ^^ Language: ATL0.1 0.2 0.3 0.4 0.5 0.6Normalized brain alignment *^ ^ Language: I...

    2025

  7. [1999]

    What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, pp

    Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, pp. 2022–03,

    2022

  8. [2014]

    Transformers: State-of-the-art natural language processing

    14 Published as a conference paper at ICLR 2025 Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings o...

    2025

  9. [2016]

    Gallant lab natural short clips 3t fmri data

    12 Published as a conference paper at ICLR 2025 Alexander G Huth, Shinji Nishimoto, An T Vu, and T Dupre La Tour. Gallant lab natural short clips 3t fmri data. G-Node doi, 10,

    2025

  10. [2017]

    Vision-and-language or vision-for- language? on cross-modal influence in multimodal transformers

    Stella Frank, Emanuele Bugliarello, and Desmond Elliott. Vision-and-language or vision-for- language? on cross-modal influence in multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9847–9857,

    2021

  11. [2018]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557,

    1908 arXiv

  12. [2019]

    Proper and common names in the semantic system

    2https://docs.cneuromod.ca/en/latest/ACCESS.html 11 Published as a conference paper at ICLR 2025 Rutvik H Desai, Usha Tadimeti, and Nicholas Riccardi. Proper and common names in the semantic system. Brain Structure and Function, 228(1):239–254,

    2025

  13. [2021]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413,

    arXiv

  14. [2022]

    Interpreting multimodal video transformers using brain recordings

    Dota Tianai Dong and Mariya Toneva. Interpreting multimodal video transformers using brain recordings. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023a. Dota Tianai Dong and Mariya Toneva. Vision-language integration in multimodal video tr...

    2023 arXiv

  15. [2023]

    Mae-ast: Masked autoencoding audio spectrogram transformer

    Alan Baade, Puyuan Peng, and David Harwath. Mae-ast: Masked autoencoding audio spectrogram transformer. Interspeech 2022,

    2022

  16. [2024]

    What aspects of nlp models and brain datasets affect brain-nlp alignment? In 2023 Conference on Cognitive Computational Neuroscience,

    Subba Reddy Oota and Mariya Toneva. What aspects of nlp models and brain datasets affect brain-nlp alignment? In 2023 Conference on Cognitive Computational Neuroscience,

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.