Pith. sign in

REVIEW 3 major objections 8 minor 3 cited by

TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single multimodal model predicts whole-brain fMRI responses to movies across subjects.

desk verdict Solid competition-winning encoding model with a real but addressable validation-selection concern. read the letter →

arxiv 2507.22229 v1 pith:LEGCY23A submitted 2025-07-29 cs.LG

Stéphane d'Ascoli , Jérémy Rapin , Yohann Benchetrit , Hubert Banville , Jean-Rémi King This is my paper · ORCID
classification cs.LG
keywords brainencodingfMRImultimodaltransformernaturalisticmovieswhole-brainpredictionmulti-subjectlearningmodalitydropout
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TRIBE, a deep encoding pipeline that predicts the brain's BOLD responses to naturalistic videos from three input streams at once: the transcript, the audio, and the video frames. It combines frozen representations from a language model, a speech model, and a video model, then lets a transformer integrate them over time. The central claim is that this is the first encoding pipeline that is simultaneously nonlinear, multisubject, and multimodal, and that this combination is what lets a single model reach top accuracy across 1,000 cortical parcels and four participants. If the claim holds, whole-brain encoding no longer needs one linear model per modality per subject, and the parts of the brain that integrate senses can be studied with a predictive model rather than by comparing separate unimodal fits.

What carries the argument

The load-bearing design is a transformer encoder fed with time-aligned embeddings from three frozen foundation models: Llama-3.2-3B for transcript words, Wav2Vec-Bert-2.0 for audio, and V-JEPA-2 for video, all resampled to a common 2 Hz grid. Each modality's intermediate layers are grouped and averaged, projected to a shared dimension, concatenated, and given learnable positional and subject embeddings. The transformer exchanges information across time, an adaptive pooling layer compresses the sequence to one embedding per fMRI repetition time, and a subject-conditional linear layer maps the result onto 1,000 parcels. During training, modality dropout randomly masks whole modalities so the model learns to predict from any subset; the final predictions come from a per-parcel softmax ensemble over 1,000 models.

What would settle it

Re-run the pipeline with the ensemble weights and early-stopping decisions fixed on a split that is never used during development, and evaluate on a fresh held-out set; if the mean test correlation falls below the runner-up's score, the reported margin is an artifact of validation-based selection. Separately, restrict the noise-ceiling estimate to recording sessions that are disjoint from the sessions used to compute the model's correlations; a normalized Pearson well below 0.54 would show that the explainable-variance claim is inflated by session overlap.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that combining text, audio, and video embeddings in one transformer-based encoder improves fMRI prediction over any unimodal encoder, and that the gains are largest in associative cortices such as prefrontal and parieto-occipito-temporal areas, while the best unimodal model still wins in primary visual cortex. The model reaches a mean Pearson correlation of 0.2146 on the competition's out-of-distribution held-out movies and 0.3195 on an in-distribution Friends episode, and it would capture about 54% of the noise-ceiling-normalized explainable variance. The paper also reports that removing the transformer drops the score from 0.31 to 0.23 and removing multi-subject training drops it to 0.29, and that adding either more recording sessions or longer language context keeps improving performance without plateauing.

Load-bearing premise

The quantitative results, including the winning margin and the 54% of explainable variance, depend on the competition's held-out test scores being a faithful measure of generalization, while the model's early stopping, hyperparameter sampling, and per-parcel ensemble weights are all chosen on the validation set.

Editorial extensions

If this is right

  • Whole-brain encoding becomes a single-model problem: the same nonlinear, multimodal network can serve multiple subjects and all cortical parcels, removing the need to fit per-subject, per-modality ridge regressions.
  • Multisensory integration is where multimodality pays off: the largest gains are in association cortices, so encoding models aimed at those areas should not be built from a single modality.
  • Primary sensory areas still prefer their own modality, so unimodal and multimodal models are complementary rather than interchangeable.
  • Recording more data and giving the language model a longer context both improve predictions on a trend that has not plateaued, which predicts further gains from larger datasets and longer contexts.
  • Because modality dropout lets the model run with missing inputs, the same pipeline can be applied to silent film, audio-only, or text-only stimuli without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's spatial averaging of video patch tokens is identified by the authors themselves as a likely cost in retinotopic visual areas; a version that keeps patch positions should specifically improve low-level visual parcels, a test the paper's data makes possible.
  • The per-parcel softmax ensemble over 1,000 models suggests that different parcels may benefit from different hyperparameters; a single model with per-parcel readout heads might capture part of that benefit at a fraction of the training cost.
  • Because the frozen feature extractors are never fine-tuned, the approach predicts that further gains are available by adapting those representations to the brain-encoding objective, at the risk of overfitting the four available subjects.
  • The same architecture should transfer to other naturalistic recording settings, such as audiobooks or virtual-reality navigation, where text, audio, and video streams can be time-aligned in the same way.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The manuscript introduces TRIBE, a transformer-based encoding model that predicts whole-brain fMRI responses from text, audio, and video features extracted from frozen pretrained models (Llama-3.2, Wav2Vec-Bert-2.0, V-JEPA 2). The model is trained on four subjects of the Courtois NeuroMod dataset and evaluated on the Algonauts 2025 competition, where it reports first place. The paper includes modality ablation analyses showing gains from multimodality, a noise-ceiling normalization reporting 54% of explainable variance, and scaling-law experiments. Central claims are that the pipeline is the first simultaneously nonlinear, multisubject, and multimodal encoder, and that it outperforms competitors by a significant margin.

Significance. The external competition result is strong evidence that the pipeline works in practice, and the code is released. If the statistical concerns below are addressed, this would be a valuable contribution to the brain-encoding literature. The combination of three modalities with a nonlinear multisubject model is practically useful, and the per-parcel maps of multimodal gains are potentially interesting to neuroscientists. However, the headline margin and the 54% explainable-variance fraction need stronger statistical support before the claims can be taken at face value.

major comments (3)
  1. [Section 2.4 and Table 1] The per-parcel ensemble selection over M=1000 models using validation Pearson scores and softmax temperature 0.3, combined with early stopping and hyperparameter sampling on the same validation set, is a high-capacity selection procedure; the manuscript reports no error bars or significance tests for the leaderboard gap, so the word 'significant' in the Abstract and Section 3.1 is not statistically supported, especially because the subject-5 margin is only 0.0002 (0.1720 vs 0.1718). Please provide a bootstrap over subjects or sessions, or at least report per-subject standard errors, and discuss whether the validation-based ensemble selection transfers to the test set.
  2. [Section 3.2, Eq. (1)] The manuscript does not state whether the model correlation ρ in Eq. (1) is computed on held-out data disjoint from the repeated-viewing movies (Hidden Figures, Life) used to estimate ρ_self; if the same sessions contribute to both quantities, the reported 54% of explainable variance is inflated because ρ and ρ_self are not independent. Please specify the exact data split used for ρ and ρ_self and, if necessary, recompute the normalized scores on disjoint sessions.
  3. [Section 4 and Abstract] The claim that TRIBE is 'the first encoding pipeline which is simultaneously nonlinear, multisubject and multimodal' is not established by the cited related work; in particular, the Deep Recurrent Encoder [19] is cited as relaxing linearity and may already be multisubject, so the novelty should be scoped to the specific combination of three input modalities and a transformer encoder, rather than to the individual properties. Please temper or justify this claim with a more systematic comparison.
minor comments (8)
  1. [Section 3.1 and Table 1] The text reports 'out of 262 teams' while the Table 1 caption says 'out of 263 teams'; please reconcile the two numbers.
  2. [Section 2.1] The phrase 'start-of-the art' should be 'state-of-the-art'.
  3. [Figure 5b caption] The word 'substracting' should be 'subtracting'.
  4. [Section 3.4] The sentence 'text+audio (yellow) in can be observed in the superior temporal lobe' contains a grammatical error ('in' is repeated); please revise.
  5. [Section 2.3] The audio embeddings are noted to carry bidirectional information while text and video embeddings are causal; this asymmetry is not discussed in the modality ablation results and could affect their interpretation.
  6. [Table 3] The 'Layer groups' hyperparameter values such as [0.5, 0.75, 1] are not defined; please specify whether these are fractions of the total number of layers or absolute layer indices.
  7. [Figure 1] The Pearson correlation values printed next to the brain parcels are not defined in the caption; please state that they are the correlation between predicted and actual BOLD for the first participant and the first held-out movie.
  8. [General] The code repository is mentioned only in the Abstract; please add a Code Availability section in the main text or at the end of the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: TRIBE's performance is measured against an external competition leaderboard and no fitted parameter is renamed as a prediction.

full rationale

The central claims are empirical: the model maps frozen text/audio/video features through a learned transformer to BOLD responses, and its performance is assessed on held-out Algonauts 2025 test movies (Tables 1-2) rather than on quantities fit during training. The noise-ceiling normalization (Eq. 1) is an external standardization from Schoppe et al. and does not make the model's correlation equal to its own input; at worst, an ambiguous choice of sessions could bias the 0.54 number, which is a measurement validity concern, not circularity. The per-parcel softmax ensembling (Section 2.4) fits weights to validation scores, so validation performance is expected to improve by construction, but the reported leaderboard scores are on independent test data; this is a potential overfitting risk, not a reduction of test results to validation inputs. The self-citations ([14], [19], [42]) are used for contextual claims or an architectural choice ('Following [42]') and are not load-bearing evidence for the paper's main result; no uniqueness theorem is imported and no known result is merely renamed. Consequently, no step in the claimed derivation chain is equivalent to its inputs by construction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical engineering benchmark; it introduces no new physical or mathematical entities. Its central claim depends on a set of hand-chosen hyperparameters (context length, dropout, ensemble temperature, window size, feature frequency, layer grouping) and on the validity of using pretrained foundation model features as inputs. These are listed above as free parameters and axioms.

free parameters (7)
  • Context length k = 1024 words
    Number of preceding transcript words prepended to each embedded word; Figure 6c shows performance still rising at 1024, so the chosen value is a hand-chosen resource limit and directly affects text encoding quality.
  • Modality dropout probability p = 0.2 (grid 0.0, 0.4 for ensembling)
    Probability of masking each modality during training; hand-chosen, affects robustness and the unimodal-vs-multimodal comparison.
  • Ensemble temperature = 0.3
    Softmax temperature for per-parcel weighting of 1,000 models based on validation scores; hand-chosen, affects final test scores.
  • Window length = 100 TRs (~149 s), jitter 10 s
    Transformer context window length in TRs; hand-chosen, affects temporal integration.
  • Temporal feature frequency f = 2 Hz
    Grid frequency for all three feature streams; hand-chosen, defines the alignment between modalities and the TR grid.
  • Layer groups = [0.5, 0.75, 1] of layers, with grid variants
    Fractional layer grouping for compressing intermediate layers per modality; hand-chosen, affects which depth information is retained.
  • Transformer hidden size / depth = D=3072 hidden, 8 layers, 8 heads
    Architecture hyperparameters chosen by hand, not fitted to data; central to model capacity.
assumptions (5)
  • standard math Standard transformer attention, layer normalization, and backpropagation are assumed.
    Used throughout Section 2.3-2.4 without proof.
  • domain assumption BOLD signals at 3T, TR=1.49s, parcellated into 1,000 Schaefer parcels and z-scored per session, are a valid and sufficient target for modeling brain responses.
    Section 2.2 defines the target; if parcel-level z-scored BOLD is not a faithful readout, the central predictions are meaningless.
  • domain assumption Pretrained features from Llama-3.2-3B, Wav2Vec-Bert-2.0, and V-JEPA-2-Gigantic overlap with brain representations sufficiently to be mapped to BOLD.
    Section 2.3 relies on this alignment, supported by prior work [12-15] but not demonstrated for this combination.
  • ad hoc to paper The custom timed-text embedding construction (prepend 1,024 words, average overlapping tokens, sum into 2Hz bins) preserves the semantic timing information needed for prediction.
    Section 2.3 introduces this preprocessing without independent validation.
  • ad hoc to paper Spatially averaging video patch tokens discards position information but retains enough visual information for whole-brain prediction.
    Section 2.3, Video embeddings: 'this spatial averaging step was necessary... at the cost of discarding positional information'; the paper accepts this tradeoff.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction." pith.science (2026). https://pith.science/paper/LEGCY23A

@misc{pith2026250722229,
  author       = {Pith},
  title        = {Pith review of: TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEGCY23A}},
  note         = {Machine review of arXiv:2507.22229}
}
read the original abstract

Historically, neuroscience has progressed by fragmenting into specialized domains, each focusing on isolated modalities, tasks, or brain regions. While fruitful, this approach hinders the development of a unified model of cognition. Here, we introduce TRIBE, the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas and individuals. By combining the pretrained representations of text, audio and video foundational models and handling their time-evolving nature with a transformer, our model can precisely model the spatial and temporal fMRI responses to videos, achieving the first place in the Algonauts 2025 brain encoding competition with a significant margin over competitors. Ablations show that while unimodal models can reliably predict their corresponding cortical networks (e.g. visual or auditory networks), they are systematically outperformed by our multimodal model in high-level associative cortices. Currently applied to perception and comprehension, our approach paves the way towards building an integrative model of representations in the human brain. Our code is available at https://github.com/facebookresearch/algonauts-2025.

Figures

Figures reproduced from arXiv: 2507.22229 by the authors.

Figure 1
Figure 1. TRIBE predicts brain responses to videos across diverse regions. For eight brain parcels color-coded in the brain images, we report the BOLD response of the first participant to the first 5 minutes of a held-out movie in solid lines and our model’s predictions in dashed lines, with the Pearson correlation of the two curves reported on the left. First, linearity: existing encoding approaches typically rely on a ridge… view at source ↗
Figure 2
Figure 2. Visual summary of our method. 2 Methods 2.1 Overview Task Our objective is to predict the brain activity of participants watching videos. This is framed as a regression task where the targets are the blood-oxygen-level-dependent (BOLD) signals detected at every repetition time (TR=1.49s) of a 3T fMRI recording device, separated into 1,000 non-overlapping parcels fig. 1. For this, we take as input the video clip bein… view at source ↗
Figure 3
Figure 3. Our model performs consistently across the whole brain. (a) We report a histogram of unnormalized and normalized encoding scores (see definition in eq. (1)) of the 1000 parcels, averaged across subjects. (b) We project the normalized encoding scores of the first subject onto the fsaverage5 cortical surface. 3.2 Noise ceiling analysis Following common practice in the brain encoding literature [47], we estimate the no… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Multimodal encoders outperform unimodal encoders. (a) We compare the encoding scores of encoders trained on a subset of modalities in the same conditions. Error bars denote s.e.m across subjects. (b) For each parcel of the first subject, we compute the difference in en…
Figure 5
Figure 5. Figure 5: The modalities and their interplay map onto the expected brain areas. (a) For each parcel of the first subject, we assess the encoding contribution of each modality by masking the two other modalities, and color-code according to the highest contribution. (b) We color-…
Figure 6
Figure 6. Figure 6: Ablations and scaling laws of our model. (a) We compare the results of our model with those achieved without multi-subject training or without the transformer model. (b) We report the results obtained by the multimodal encoder as we increase the number of sessions used…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NeuroWorld: A Latent Brain World Model for Stimulus-Conditioned Human Brain Dynamics

    q-bio.NC 2026-08 conditional novelty 6.0 of 10

    A latent world model of the brain, trained to predict the next latent fMRI state from past brain states and current movie stimuli, outperforms regression-based encoders in causal multi-step rollout on three naturalist...

  2. Predicted Cortex Is Not a Domain-General Prior: A Matched-Control Audit of Brain-Encoding Features for Video Memorability

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Predicted cortical responses from a brain-encoding model beat their own visual backbone on VideoMem but lose on Memento10k, so they are a dataset-specific memorability representation, not a domain-general prior.

  3. A global predicted-fMRI drive signal from TRIBE does not predict YouTube replay heatmaps

    cs.SE 2026-07 unverdicted novelty 6.0 of 10

    TRIBE’s predicted cortical drive does not predict YouTube most-replayed heatmaps beyond position and low-level baselines, with the null bounded near r≈0.14.

Reference graph

Works this paper leans on

49 extracted references · 35 canonical work pages · cited by 3 Pith papers

  1. [19]

    Deep recurrent encoder: A scalable end-to-end network to model brain signals

    Omar Chehab, Alexandre Defossez, Jean-Christophe Loiseau, Alexandre Gramfort, and Jean- Remi King. Deep recurrent encoder: A scalable end-to-end network to model brain signals. arXiv preprint arXiv:2103.02339, 2021

    arXiv 2021

  2. [1]

    Neural basis of a perceptual decision in the parietal cortex (area lip) of the rhesus monkey

    Michael N Shadlen and William T Newsome. Neural basis of a perceptual decision in the parietal cortex (area lip) of the rhesus monkey. Journal of neurophysiology, 86(4):1916–1936, 2001

    work page 1916

  3. [2]

    The fusiform face area: a cortical region specialized for the perception of faces

    Nancy Kanwisher and Galit Yovel. The fusiform face area: a cortical region specialized for the perception of faces. Philosophical Transactions of the Royal Society B: Biological Sciences, 361(1476):2109–2128, 2006

    work page 2006

  4. [3]

    The unique role of the visual word form area in reading

    Stanislas Dehaene and Laurent Cohen. The unique role of the visual word form area in reading. Trends in cognitive sciences, 15(6):254–262, 2011

    work page 2011

  5. [4]

    Conscious processing and the global neuronal workspace hypothesis

    George A Mashour, Pieter Roelfsema, Jean-Pierre Changeux, and Stanislas Dehaene. Conscious processing and the global neuronal workspace hypothesis. Neuron, 105(5):776–798, 2020

    work page 2020

  6. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [6]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [7]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    arXiv 2023

Show all 49 references
  1. [8]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020

    2020

  2. [9]

    W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training

    Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)...

    2021

  3. [10]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022

    2022

  4. [11]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025

    2025 arXiv

  5. [12]

    Performance-optimized hierarchical models predict neural responses in higher visual cortex

    Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619–8624, 2014

    2014

  6. [13]

    Natural speech reveals the semantic maps that tile human cerebral cortex

    Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453–458, 2016

    2016

  7. [14]

    Brains and algorithms partially converge in natural language processing

    Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022

    2022

  8. [15]

    Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018

    Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018. 10

    2018

  9. [16]

    Toward a realistic model of speech processing in the brain with self-supervised learning

    Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et al. Toward a realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems, 35:334...

    2022

  10. [17]

    Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

    Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32, 2019

    2019

  11. [18]

    Can deep neural networks learn biological vision? arXiv preprint arXiv:2504.16940, 2025

    Drew Linsley, Pinyuan Feng, and Thomas Serre. Can deep neural networks learn biological vision? arXiv preprint arXiv:2504.16940, 2025

    2025 arXiv

  12. [20]

    Neural processing of naturalistic audiovisual events in space and time

    Yu Hu and Yalda Mohsenzadeh. Neural processing of naturalistic audiovisual events in space and time. Communications Biology, 8(1):110, 2025

    2025

  13. [21]

    Audiovisual integration in the human brain: a coordinate-based meta-analysis

    Chuanji Gao, Jessica J Green, Xuan Yang, Sewon Oh, Jongwan Kim, and Svetlana V Shinkareva. Audiovisual integration in the human brain: a coordinate-based meta-analysis. Cerebral Cortex, 33(9):5574–5584, 2023

    2023

  14. [22]

    See me, hear me, touch me: multisensory integration in lateral occipital- temporal cortex

    Michael S Beauchamp. See me, hear me, touch me: multisensory integration in lateral occipital- temporal cortex. Current opinion in neurobiology, 15(2):145–153, 2005

    2005

  15. [23]

    Multisensory interplay reveals crossmodal influences on ‘sensory-specific’brain regions, neural responses, and judgments

    Jon Driver and Toemme Noesselt. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’brain regions, neural responses, and judgments. Neuron, 57(1):11–23, 2008

    2008

  16. [24]

    Multisensory integration: current issues from the perspective of the single neuron

    Barry E Stein and Terrence R Stanford. Multisensory integration: current issues from the perspective of the single neuron. Nature reviews neuroscience, 9(4):255–266, 2008

    2008

  17. [25]

    Sim: Surface-based fmri analysis for inter-subject multimodal decoding from movie-watching experiments

    Simon Dahan, Gabriel Bénédict, Logan Zane John Williams, Yourong Guo, Daniel Rueckert, Robert Leech, and Emma Claire Robinson. Sim: Surface-based fmri analysis for inter-subject multimodal decoding from movie-watching experiments. In The Thirteenth International Conference on ...

    2025

  18. [26]

    Umbrae: Unified multi- modal brain decoding

    Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Umbrae: Unified multi- modal brain decoding. In European Conference on Computer Vision, pages 242–259. Springer, 2024

    2024

  19. [27]

    Modeling the dynamics of human brain activity with recurrent neural networks

    Umut Güçlü and Marcel AJ Van Gerven. Modeling the dynamics of human brain activity with recurrent neural networks. Frontiers in computational neuroscience, 11:7, 2017

    2017

  20. [28]

    Vision-language integration in multimodal video transformers (partially) aligns with the brain

    Dota Tianai Dong and Mariya Toneva. Vision-language integration in multimodal video transformers (partially) aligns with the brain. arXiv preprint arXiv:2311.07766, 2023

    2023 arXiv

  21. [29]

    Visio- linguistic brain encoding

    Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S Bapi. Visio- linguistic brain encoding. arXiv preprint arXiv:2204.08261, 2022

    2022 arXiv

  22. [30]

    Semantic scene descriptions as an objective of human vision

    Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209.11737, 10, 2022

    2022 arXiv

  23. [31]

    Incorporating natural language into vision models improves prediction and understanding of higher visual cortex

    Aria Y Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv, pages 2022–09, 2022

    2022

  24. [32]

    Brain encoding models based on multimodal transformers can transfer across language and vision

    Jerry Tang, Meng Du, Vy V o, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in neural information processing systems, 36:29654–29666, 2023. 11

    2023

  25. [33]

    Botvinick, Andrew Zisserman, Oriol Vinyals, and Jo¯ao Carreira

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and Jo¯ao Carreira. Perceiver IO: A General A...

    2021

  26. [34]

    Omnivec2 - a novel transformer based network for large scale multimodal and multitask learning

    Siddharth Srivastava and Gaurav Sharma. Omnivec2 - a novel transformer based network for large scale multimodal and multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27412–27424, June 2024

    2024

  27. [35]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Ol...

    2024

  28. [36]

    cneuromod-things: a large-scale fmri dataset for task-and data-driven assessment of object representation and visual memory recognition in the human brain

    Marie St-Laurent, Basile Pinsard, Oliver Contier, Katja Seeliger, Valentina Borghesani, Julie Boyle, Pierre Bellec, and Martin Hebart. cneuromod-things: a large-scale fmri dataset for task-and data-driven assessment of object representation and visual memory recognition in the...

    2023

  29. [37]

    The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies

    Alessandro T Gifford, Domenic Bersch, Marie St-Laurent, Basile Pinsard, Julie Boyle, Lune Bellec, Aude Oliva, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies. arXiv preprint arXiv:2501.00504, 2024

    2025 arXiv

  30. [38]

    fmriprep: a robust preprocessing pipeline for functional mri

    Oscar Esteban, Christopher J Markiewicz, Ross W Blair, Craig A Moodie, A Ilkay Isik, Asier Erramuzpe, James D Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. fmriprep: a robust preprocessing pipeline for functional mri. Nature methods, 16(1):111–116, 2019

    2019

  31. [39]

    The mni brain and the talairach atlas

    Matthew Brett. The mni brain and the talairach atlas. www. mrc-Mrc-cbu. cam. ac. uk/Imaging/mnispace. Html, 2002

    2002

  32. [40]

    Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri

    Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 28(9):3095–3114, 2018

    2018

  33. [41]

    Imaging retinotopic maps in the human brain

    Brian A Wandell and Jonathan Winawer. Imaging retinotopic maps in the human brain. Vision research, 51(7):718–737, 2011

    2011

  34. [42]

    Decoding speech perception from non-invasive brain recordings

    Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1097–1107, 2023

    2023

  35. [43]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  36. [44]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  37. [45]

    Hugging face

    Shashank Mohan Jain. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pages 51–67. Springer, 2022

    2022

  38. [46]

    Array programming with numpy

    Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825):357–362, 2020

    2020

  39. [47]

    Harper, Ben D

    Oliver Schoppe, Nicol S. Harper, Ben D. B. Willmore, Andrew J. King, and Jan W. H. Schnupp. Measuring the performance of neural models.Frontiers in Computational Neuroscience, V olume 10 - 2016, 2016. 12

    2016

  40. [48]

    Scaling laws for language encoding models in fmri

    Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36:21895–21907, 2023

    2023

  41. [49]

    Computational language modeling and the promise of in silico experimentation

    Shailee Jain, Vy A V o, Leila Wehbe, and Alexander G Huth. Computational language modeling and the promise of in silico experimentation. Neurobiology of Language, 5(1):80–106, 2024. Appendices A Hyperparameters Hyperparameter Base value Other values used for ensembling Number ...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.