Pith. sign in

REVIEW 2 major objections 5 minor 75 references

SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A surface vision transformer, aligned across subjects by functional registration and trained with tri-modal contrastive learning, can retrieve which movie clip someone was watching from cortical fMRI alone—even for people and clips never…

desk verdict Good retrieval results on held-out subjects and clips, but the abstract's 'new movies' claim goes beyond the experimental design; the core method is sound and worth reviewing. read the letter →

arxiv 2501.16471 v1 pith:ZBDFKHN2 submitted 2025-01-27 cs.LG cs.AIeess.ASeess.IVq-bio.NC

Simon Dahan , Gabriel Bénédict , Logan Z. J. Williams , Yourong Guo , Daniel Rueckert , Robert Leech , Emma C. Robinson This is my paper · ORCID
classification cs.LGcs.AIeess.ASeess.IVq-bio.NC
keywords surfacevisiontransformerfMRIdecodingcross-subjectgeneralizationcontrastivelearningmovie-watchingfunctionalalignmentmultimodalretrievalcorticalanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a surface vision transformer can retrieve which 3-second movie clip a person was watching purely from their cortical fMRI, and that this retrieval works for viewers and clips the model never encountered during training. The method maps each subject's fMRI onto a spherical cortical surface via functional alignment, then uses contrastive learning to align fMRI, video, and audio embeddings in a shared space. On 7T movie-watching data from 174 participants, the model reaches 76.8% top-1 accuracy for new subjects watching clips seen in training, and 19.2% top-1 accuracy for both new subjects and new clips, against a 4.5% random baseline. If the result holds, brain-computer interfaces and neurofeedback could pool data across people and respond to novel stimuli without per-subject calibration.

What carries the argument

The load-bearing mechanism is the surface vision transformer (SiT) applied to an icosahedral discretization of the cortical surface, where the cortex is treated as a 'moving image' across time. Each subject's fMRI is resampled onto the I6 icosphere (40,962 vertices) and patched into 1,280 triangular tokens; the transformer learns long-range spatial interactions between these tokens. This encoding is pre-trained by a video masked autoencoder (vsMAE) task and then mapped, through trainable projectors, into a shared space with frozen VideoMAE and wav2vec 2.0 embeddings via a six-directional CLIP contrastive loss. Cross-subject transfer is enabled by MSMAll functional alignment, which brings functional activations from different brains into correspondence on the sphere before any model is trained.

What would settle it

Train the same SIM pipeline on HCP movie clips and test on held-out movie clips from a substantially different genre or visual style not present in the training set (e.g., animation or non-English films). If top-1 retrieval on new subjects and new clips falls to roughly the 4–5% random level, the claimed generalization to unseen stimuli is limited to the distribution of the training movies rather than a general audiovisual decoder.

Watch

Extended reading notes

Core claim

The central discovery is that cortical fMRI, treated as a spatio-temporal signal on a spherical surface mesh and encoded by a surface vision transformer, carries enough information to identify the audiovisual stimulus a person is watching, and that this mapping transfers across individuals after functional alignment. SIM first projects each subject's fMRI to a standard icosahedral sphere, patches it into 1280 triangular tokens, and pre-trains the transformer to reconstruct masked fMRI frames (vsMAE). The resulting encoder is then aligned by a tri-modal CLIP loss with frozen video and audio encoders, so embeddings of fMRI, video, and audio from the same 3-second clip are pulled together. At test time, retrieval is performed by ranking candidate clips by cosine similarity between their embeddings and the fMRI embedding. Experiments 1–3 successively remove training subjects and training clips; the framework retains significant retrieval performance in both removal settings. Attention-map analysis shows the model's heads specialize in visual and sensorimotor/auditory networks, and the same embeddings support frame-level video reconstruction from a single training subject.

Load-bearing premise

The argument depends on MSMAll functional alignment making different people's cortical fMRI comparable enough that a single transformer trained on 124 subjects applies to a new subject with no per-subject calibration; if this alignment is incomplete for an individual, the cross-subject retrieval results would not generalize.

Editorial extensions

If this is right

  • New users of a decoding system would not need tens of hours of personal fMRI recordings: a model pre-trained on a group can be applied directly, within minutes of scanning.
  • Because video and audio jointly improve retrieval beyond either modality alone, multimodal stimulus design can sharpen decoding and encoding of brain responses.
  • The same CLIP-aligned embeddings support inverse retrieval (video-to-fMRI and audio-to-fMRI), so the model can predict which cortical activations a novel stimulus would evoke.
  • Attention heads in the fMRI transformer concentrate on the visual and sensorimotor/auditory networks identified in prior brain atlases, offering a window into which systems drive decoding.
  • A regression-plus-diffusion pipeline fed by the fMRI embeddings reconstructs recognizable video frames for unseen subjects and unseen scenes, pointing toward content-level reconstructions rather than mere clip identification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, one could test whether the same cross-subject alignment transfers to patient populations or children; the current data are healthy adults, so the method's robustness to atypical cortical anatomy is unmeasured.
  • The 3-second clip length and 6-second haemodynamic lag were tuned for this dataset; extending the window to tens of seconds might let the model capture memory or attentional dynamics the authors did not explore.
  • Because the model trains on a limited set of Hollywood movie scenes, a reader should expect that retrieval performance on very different stimulus distributions (documentaries, animation, non-English cinema) is an open question, not a result.
  • If attention maps are subject-specific, they could be used to predict individual behavioural or cognitive traits; the authors flag this as future work, so treating it as a consequence would be over-reading.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper introduces SIM, a surface-based vision transformer framework for decoding movie stimuli from HCP 7T fMRI. The method pre-trains a surface vision transformer (SiT) encoder with a video masked autoencoder, then aligns fMRI, video, and audio embeddings via a tri-modal CLIP contrastive loss. The authors evaluate retrieval of video and audio clips from fMRI under three experimental setups: new subjects on seen clips (Experiment 1), new clips from seen movies on training subjects (Experiment 2), and new clips on new subjects (Experiment 3). They report strong top-1 retrieval accuracies, show that attention maps correlate with known functional networks, and provide qualitative video-frame reconstructions from fMRI.

Significance. If the generalization claims are properly scoped, this is a valuable contribution. The paper demonstrates that a surface transformer combined with functional alignment (MSMAll) can support cross-subject decoding without per-subject calibration, and that frozen externally pretrained video and audio encoders can be aligned with fMRI embeddings. The quantitative improvements over ridge regression are large (e.g., 76.8% vs 15.6% top-1 in Experiment 1; 19.2% vs 5.2% in Experiment 3). The paper also provides detailed experimental appendices and promises code and pretrained models. However, the central advertised claim that decoding works for movies not seen during training is not supported by the experiments, and the random baseline reporting needs correction. These issues are fixable within the manuscript's scope.

major comments (2)
  1. [Abstract; Sections 4.4 and 6] The abstract states that decoding works 'even for individuals and movies not seen during training', but this is not tested. Experiments 2 and 3, described in Section 4.4 and Figure 2, train on the first half of each movie and test on the last half of the same movies; the test clips therefore share narrative, characters, and audio-visual style with the training clips. The paper's own Discussion (Section 6) acknowledges 'we were unable to effectively test whether the model would generalise to completely different movies.' The central claim should be revised to 'new clips from movies seen during training' throughout the abstract, introduction, and Section 5.
  2. [Table 1 and Section 4.3] The random baseline for video retrieval appears inconsistent with the stated candidate count. Section 4.3 states that M = 64 for video and M = 32 for audio testing; with M = 64 candidates, random top-1 accuracy should be 1.56%, yet Table 1 reports 3.7% for the fMRI-to-video random baseline, and Table C.1 reports similar values. Please clarify the exact candidate counts used for each random baseline and ensure the reported random accuracies match 1/M.
minor comments (5)
  1. [Section 4.1] Typo: '24 measured msotion parameters' should read '24 measured motion parameters'.
  2. [Figure 1 caption] Typo: 'projected to vector spaces of common lenght' should read 'common length'.
  3. [Figure 5 caption] The caption says 'Negative pairs (31)', but the text uses M = 32 candidates, so the number of negative pairs is M - 1 = 31; please state this explicitly.
  4. [Section 4.2] Please clarify how the 200 validation and 200 testing samples are obtained: presumably 25 subjects x 2 hemispheres x 4 movie sessions = 200, and similarly for validation.
  5. [Sections 4.4 and 5] The phrase 'new movie scenes' is used in Figure 2 and the text to describe Experiments 2 and 3, but the experiments only test new clips from the later halves of the same movies; consider renaming this to 'new clips from seen movies' to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out retrieval, external frozen encoders, and in-paper ablations make the central decoding claim self-contained.

full rationale

The central retrieval claim is not derived from its own inputs. Experiment 1 (Sec. 4.4, Fig. 2(i)) trains on 124 subjects and tests on 25 held-out subjects with the same clips; Experiment 3 (Fig. 2(iii)) trains on the first half of each movie and tests on the last half, so the test clips are distinct 3s intervals whose fMRI and video/audio embeddings were never seen during training. The fMRI-to-video retrieval numbers (76.8% top-1 hard-negative in Table 1; 19.2% top-1 soft-negative in Experiment 3, Table C.3) are evaluated against random and Ridge baselines, so they cannot be fitted values renamed as predictions. Video and audio encoders are frozen and pretrained on external corpora (Kinetics, LibriSpeech; Sec. B.3), and the vsMAE ablation in Table C.1 shows the contribution of pre-training within the paper's own data. The self-citations (SiT, Dahan et al. 2022; vsMAE, Dahan et al. 2024) supply architecture and a pre-training recipe, not the decoding result, and their role is independently checked by the paper's Ridge comparison. The one genuine caveat is scope, not circularity: Sec. 6 states 'we were unable to effectively test whether the model would generalise to completely different movies,' and Experiments 2/3 hold out only the second half of the same movies, so the abstract's 'movies not seen during training' phrasing overstates the tested generalization. That is an evidence/claim gap, not a reduction of a result to its inputs, and it does not affect the validity of the held-out clip-level retrieval evaluation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. It relies on several domain assumptions: functional alignment across subjects, spherical surface representation, external stimulus embeddings, and a fixed hemodynamic lag. It also selects a few hyperparameters on validation data, which should be read as fitted choices rather than independent predictions.

free parameters (4)
  • Masking ratio rho = 0.5
    Selected from validation MSE over {0.25, 0.5, 0.75, 0.9} on HCP fMRI reconstruction (Appendix C.7); affects the vsMAE encoder used for decoding.
  • Temporal lag tau = 6 seconds
    Selected by ridge regression correlation of video latents to fMRI across lags {1, 3, 6, 10} on MOVIE1-3 predicting MOVIE4 (Appendix C.8).
  • Movie clip duration = 3 seconds
    Chosen as average movie shot duration and as the window that gave best vsMAE reconstruction (Section 4.2, Appendix A.2); determines the alignment unit.
  • CLIP temperature tau = not reported
    Inverse temperature in the contrastive softmax is standard in CLIP but not specified numerically in the paper.
assumptions (4)
  • domain assumption MSMAll functional alignment makes cortical fMRI comparable across subjects.
    Section 4.1 states data is aligned using MSMAll functional alignment, which improves overlap of functional activations across brains. Inter-subject generalization depends on this.
  • domain assumption Cortical fMRI can be represented as a moving image on a spherical icosahedral mesh.
    Section 3.1 treats cortical signals as functions on an I6 sphere and patches them into triangular tokens; this assumes surface topography preserves functional structure.
  • domain assumption VideoMAE (Kinetics) and wav2vec2.0 (LibriSpeech) embeddings contain stimulus features aligned with brain representations.
    Section B.3 extracts latent representations from these frozen pre-trained models and uses them as alignment targets for fMRI.
  • domain assumption A fixed hemodynamic lag of 6 seconds applies across subjects and movie clips.
    Section 4.2 and Appendix C.8 set the fMRI temporal lag to 6 seconds based on validation correlations, and this single lag is used for all test data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments." pith.science (2026). https://pith.science/paper/ZBDFKHN2

@misc{pith2026250116471,
  author       = {Pith},
  title        = {Pith review of: SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBDFKHN2}},
  note         = {Machine review of arXiv:2501.16471}
}
read the original abstract

Current AI frameworks for brain decoding and encoding, typically train and test models within the same datasets. This limits their utility for brain computer interfaces (BCI) or neurofeedback, for which it would be useful to pool experiences across individuals to better simulate stimuli not sampled during training. A key obstacle to model generalisation is the degree of variability of inter-subject cortical organisation, which makes it difficult to align or compare cortical signals across participants. In this paper we address this through the use of surface vision transformers, which build a generalisable model of cortical functional dynamics, through encoding the topography of cortical networks and their interactions as a moving image across a surface. This is then combined with tri-modal self-supervised contrastive (CLIP) alignment of audio, video, and fMRI modalities to enable the retrieval of visual and auditory stimuli from patterns of cortical activity (and vice-versa). We validate our approach on 7T task-fMRI data from 174 healthy participants engaged in the movie-watching experiment from the Human Connectome Project (HCP). Results show that it is possible to detect which movie clips an individual is watching purely from their brain activity, even for individuals and movies not seen during training. Further analysis of attention maps reveals that our model captures individual patterns of brain activity that reflect semantic and visual systems. This opens the door to future personalised simulations of brain function. Code & pre-trained models will be made available at https://github.com/metrics-lab/sim, processed data for training will be available upon request at https://gin.g-node.org/Sdahan30/sim.

Figures

Figures reproduced from arXiv: 2501.16471 by the authors.

Figure 1
Figure 1. (SIM) seeks to align visual and audio stimuli - extracted from T = 3 second movie clips - with brain activations acquired over the same time period: [A] 7T fMRI data, collected during movie watching, is first projected to subjects’ native cortical surfaces (resolution V =59292 vertices); then inflated to a sphere, and downsampled onto a regularly tessellated icosphere (I6 with V =40962 ver￾tices). [B] This is then e… view at source ↗
Figure 2
Figure 2. (a) Each movie-watching session (MOVIE1-4) is composed of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Video retrieval for Experiment 3 - generalisation to new movie scenes and new subjects - from: (a) soft-negative sampling: here, the reference movie clip is correctly retrieved as top1 and top-ranked movie clips all depict human faces; (b) hard-negative sampling: here the top-ranked movie clips all correspond to dialogue scenes. Generalisation to new subjects (i) Results in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Average attention maps for each attention head, extracted from the SiT encoder ( [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Soft-negative fMRI → V retrieval results for Experiment 2 and 3. Negative pairs (31) are sampled from different movies than the positive sample - but all sampled from new movie scenes (not used during training) - showing generalisation to new stimuli for train subjects…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 51 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  5. [5]

    A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

    Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci., December 2021

    work page 2021

  6. [6]

    Predictive coding or just feature discovery? an alternative account of why language models fit brain data

    Richard Antonello and Alexander Huth. Predictive coding or just feature discovery? an alternative account of why language models fit brain data. Neurobiol Lang (Camb), 5 0 (1): 0 64--79, April 2024

    work page 2024

  7. [7]

    wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020

    Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020

    2020

  8. [8]

    Brain decoding: toward real-time reconstruction of visual perception

    Yohann Benchetrit, Hubert Banville, and Jean-R \'e mi King. Brain decoding: toward real-time reconstruction of visual perception. arXiv preprint arXiv:2310.19812, 2023

    arXiv 2023

Show all 75 references
  1. [9]

    The relationship between spatial configuration and functional connectivity of brain regions

    Janine Diane Bijsterbosch, Mark W Woolrich, Matthew F Glasser, Emma C Robinson, Christian F Beckmann, David C Van Essen, Samuel J Harrison, and Stephen M Smith. The relationship between spatial configuration and functional connectivity of brain regions. Elife, 7, February 2018

    2018

  2. [10]

    Boyle J.A

    Pinsard B. Boyle J.A. The courtois project on neuronal modelling - 2020 data release. In OHBM, 2020. URL https://docs.cneuromod.ca/en/latest/DATASETS.html

    2020

  3. [11]

    Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković

    Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021. URL https://arxiv.org/abs/2104.13478

    2021 arXiv

  4. [12]

    Emerging properties in self-supervised vision transformers, 2021

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https://arxiv.org/abs/2104.14294

    2021 arXiv

  5. [13]

    Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects

    Charlotte Caucheteux, Alexandre Gramfort, and Jean-R \'e mi King. Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects. arXiv preprint arXiv:2110.06078, 2021

    2021 arXiv

  6. [14]

    Evidence of a predictive coding hierarchy in the human brain listening to speech

    Charlotte Caucheteux, Alexandre Gramfort, and Jean-R \'e mi King. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7 0 (3): 0 430--441, 2023

    2023

  7. [15]

    Very deep vaes generalize autoregressive models and can outperform them on images, 2021

    Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images, 2021. URL https://arxiv.org/abs/2011.10650

    2021 arXiv

  8. [16]

    The impact of traditional neuroimaging methods on the spatial localization of cortical areas

    TS Coalson, DC Van Essen, and MF Glasser. The impact of traditional neuroimaging methods on the spatial localization of cortical areas. Proceedings of the National Academy of Sciences of the United States of America, 115 0 (27): 0 E6356--E6365, 2018. doi:10.1073/pnas.180158211...

    2018 doi

  9. [17]

    Cutting, Kaitlin L

    James E. Cutting, Kaitlin L. Brunick, and Ayse Candan. Perceiving event dynamics and parsing hollywood films. Journal of Experimental Psychology: Human Perception and Performance, 38 0 (6): 0 1476--1490, December 2012. doi:10.1037/a0027737. URL https://doi.org/10.1037/a0027737...

    2012 doi

  10. [18]

    Surface vision transformers: Attention-based modelling applied to cortical analysis

    Simon Dahan, Abdulah Fawaz, Logan Z J Williams, Chunhui Yang, Timothy S Coalson, Matthew F Glasser, A David Edwards, Daniel Rueckert, and Emma C Robinson. Surface vision transformers: Attention-based modelling applied to cortical analysis. In Ender Konukoglu, Bjoern Menze, Arc...

    2022

  11. [19]

    Spatio-temporal encoding of brain dynamics with surface masked autoencoders

    Simon Dahan, Logan Zane John Williams, Yourong Guo, Daniel Rueckert, and Emma Claire Robinson. Spatio-temporal encoding of brain dynamics with surface masked autoencoders. In Submitted to Medical Imaging with Deep Learning, 2024. URL https://openreview.net/forum?id=9G7ZEYHLVJ....

    2024

  12. [20]

    Decoding speech perception from non-invasive brain recordings

    Alexandre D \'e fossez, Charlotte Caucheteux, J \'e r \'e my Rapin, Ori Kabeli, and Jean-R \'e mi King. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5 0 (10): 0 1097--1107, 2023

    2023

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  14. [22]

    Short-term hebbian learning can implement transformer-like attention

    Ian T Ellwood. Short-term hebbian learning can implement transformer-like attention. PLOS Computational Biology, 20 0 (1): 0 e1011843, 2024

    2024

  15. [23]

    Abdulah Fawaz, Logan Z. J. Williams, Amir Alansary, Cher Bass, Karthik Gopinath, Mariana da Silva, Simon Dahan, Chris Adamson, Bonnie Alexander, Deanne Thompson, Gareth Ball, Christian Desrosiers, Herv \'e Lombaert, Daniel Rueckert, A. David Edwards, and Emma C. Robinson. Benc...

    2021 doi

  16. [24]

    Masked autoencoders as spatiotemporal learners, 2022

    Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners, 2022

    2022

  17. [25]

    Movie-watching outperforms rest for functional connectivity-based prediction of behavior

    Emily S Finn and Peter A Bandettini. Movie-watching outperforms rest for functional connectivity-based prediction of behavior. Neuroimage, 235: 0 117963, 2021 a . doi:10.1016/j.neuroimage.2021.117963. Epub 2021 Apr 2

    2021

  18. [26]

    Movie-watching outperforms rest for functional connectivity-based prediction of behavior

    Emily S Finn and Peter A Bandettini. Movie-watching outperforms rest for functional connectivity-based prediction of behavior. Neuroimage, 235: 0 117963, July 2021 b

    2021

  19. [27]

    Freesurfer

    Bruce Fischl. Freesurfer. NeuroImage, 62: 0 774--781, 8 2012. ISSN 1095-9572. doi:10.1016/J.NEUROIMAGE.2012.01.021. URL https://pubmed.ncbi.nlm.nih.gov/22248573/

    2012

  20. [28]

    Towards building a more complex view of the lateral geniculate nucleus: Recent advances in understanding its role

    Masoud Ghodrati, Seyed-Mahdi Khaligh-Razavi, and Sidney R Lehky. Towards building a more complex view of the lateral geniculate nucleus: Recent advances in understanding its role. Progress in Neurobiology, 156: 0 214--255, 2017

    2017

  21. [29]

    Wilson, Timothy Coalson, Bruce Fischl, Jesper Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan Polimeni, Van DC, and Mark Jenkinson

    Matthew Glasser, Stamatios Sotiropoulos, J. Wilson, Timothy Coalson, Bruce Fischl, Jesper Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan Polimeni, Van DC, and Mark Jenkinson. The minimal preprocessing pipelines for the human connectome project. NeuroImage, 80: 0...

    2013 doi

  22. [30]

    A multi-modal parcellation of human cerebral cortex

    Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536 0 (7615): 0 171--178, 2016 a

    2016

  23. [31]

    The human connectome project's neuroimaging approach

    Matthew F Glasser, Stephen M Smith, Daniel S Marcus, Jesper LR Andersson, Edward J Auerbach, Timothy EJ Behrens, Timothy S Coalson, Michael P Harms, Mark Jenkinson, Steen Moeller, et al. The human connectome project's neuroimaging approach. Nature neuroscience, 19 0 (9): 0 117...

    2016

  24. [32]

    Individual-specific features of brain systems identified with resting state functional correlations

    Evan M Gordon, Timothy O Laumann, Babatunde Adeyemo, Adrian W Gilmore, Steven M Nelson, Nico UF Dosenbach, and Steven E Petersen. Individual-specific features of brain systems identified with resting state functional correlations. Neuroimage, 146: 0 918--939, 2017 a

    2017

  25. [33]

    Precision functional mapping of individual human brains

    Evan M Gordon, Timothy O Laumann, Adrian W Gilmore, Dillan J Newbold, Deanna J Greene, Jeffrey J Berg, Mario Ortega, Catherine Hoyt-Drazen, Caterina Gratton, Haoxin Sun, et al. Precision functional mapping of individual human brains. Neuron, 95 0 (4): 0 791--807, 2017 b

    2017

  26. [34]

    Accurate and robust brain image alignment using boundary-based registration

    Douglas N Greve and Bruce Fischl. Accurate and robust brain image alignment using boundary-based registration. Neuroimage, 48 0 (1): 0 63--72, 2009

    2009

  27. [35]

    Decoding natural image stimuli from fmri data with a surface-based convolutional network

    Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Decoding natural image stimuli from fmri data with a surface-based convolutional network. arXiv preprint arXiv:2212.02409, 2022

    2022 arXiv

  28. [36]

    Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies

    James V Haxby, J Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. elife, 9: 0 e56601, 2020

    2020

  29. [37]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021

    2021

  30. [38]

    THINGS -data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

    Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. THINGS -data, a multimodal collection of large-scale datasets for investigating object representations in human brain and ...

    2023

  31. [39]

    A. Huth, W. de Heer, T. Griffiths, et al. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532: 0 453--458, 2016. doi:10.1038/nature17637. URL https://doi.org/10.1038/nature17637

    2016 doi

  32. [40]

    The kinetics human action video dataset, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017

    2017

  33. [41]

    Transformer as a hippocampal memory consolidation model based on nmdar-inspired nonlinearity

    Dong Kyum Kim, Jea Kwon, Meeyoung Cha, and C Lee. Transformer as a hippocampal memory consolidation model based on nmdar-inspired nonlinearity. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Spatial topography of individual-specific cortical networks predicts human cognition, personality, and emotion

    Ru Kong, Jingwei Li, Csaba Orban, Mert R Sabuncu, Hesheng Liu, Alexander Schaefer, Nanbo Sun, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, et al. Spatial topography of individual-specific cortical networks predicts human cognition, personality, and emotion. Cerebral cortex, ...

    2019

  35. [43]

    Deep neural networks: a new framework for modeling biological vision and brain information processing

    Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1: 0 417--446, 2015

    2015

  36. [44]

    Variation in spatial dependencies across the cortical mantle discriminates the functional behaviour of primary and association cortex

    Robert Leech, Reinder Vos De Wael, František Váša, Ting Xu, R Austin Benn, Robert Scholz, Rodrigo M Braga, Michael P Milham, Jessica Royer, Boris C Bernhardt, Emily J H Jones, Elizabeth Jefferies, Daniel S Margulies, and Jonathan Smallwood. Variation in spatial dependencies ac...

    2023

  37. [45]

    Convolutional neural networks as a model of the visual system: Past, present, and future

    Grace W Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future. Journal of cognitive neuroscience, 33 0 (10): 0 2017--2031, 2021

    2017

  38. [46]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  39. [47]

    Situating the default-mode network along a principal gradient of macroscale cortical organization

    Daniel S Margulies, Satrajit S Ghosh, Alexandros Goulas, Michał Falkiewicz, Julia M Huntenburg, Georg Langs, Gleb Bezgin, Simon B Eickhoff, Francisco Xavier Castellanos, Michael Petrides, Elizabeth Jefferies, and Jonathan Smallwood. Situating the default-mode network along a p...

    2016 doi

  40. [48]

    Toward a realistic model of speech processing in the brain with self-supervised learning, 2022

    Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King. Toward a realistic model of speech processing in the brain with self-supervised learning, 2022. URL https://arxiv.org/abs/2206.01685

    2022 arXiv

  41. [49]

    Meta learning to bridge vision and language models for multimodal few-shot learning, 2023

    Ivona Najdenkoska, Xiantong Zhen, and Marcel Worring. Meta learning to bridge vision and language models for multimodal few-shot learning, 2023

    2023

  42. [50]

    narratives

    Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The “narratives” fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8...

    2021

  43. [51]

    Natural scene reconstruction from fmri signals using generative latent diffusion

    Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent diffusion. Scientific Reports, 13 0 (1): 0 15666, 2023

    2023

  44. [52]

    Librispeech: An asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 5206--5210, 2015. doi:10.1109/ICASSP.2015.7178964

    2015

  45. [53]

    Multi-dynamic modelling reveals strongly time-varying resting fmri correlations

    Usama Pervaiz, Diego Vidaurre, Chetan Gohil, Stephen M Smith, and Mark W Woolrich. Multi-dynamic modelling reveals strongly time-varying resting fmri correlations. Medical image analysis, 77: 0 102366, 2022

    2022

  46. [54]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  47. [55]

    Robinson, Saad Jbabdi, Matthew F

    Emma C. Robinson, Saad Jbabdi, Matthew F. Glasser, Jesper Andersson, Gregory C. Burgess, Michael P. Harms, Stephen M. Smith, David C. Van Essen , and Mark Jenkinson. MSM: a new flexible framework for Multimodal Surface Matching . NeuroImage, 100: 0 414--426, oct 2014. ISSN 109...

    2014

  48. [56]

    Robinson, Kara Garcia, Matthew F

    Emma C. Robinson, Kara Garcia, Matthew F. Glasser, Zhengdao Chen, Timothy S. Coalson, Antonios Makropoulos, Jelena Bozek, Robert Wright, Andreas Schuh, Matthew Webster, Jana Hutter, Anthony Price, Lucilio Cordero Grande , Emer Hughes, Nora Tusor, Philip V. Bayly, David C. Van ...

    2018

  49. [57]

    High-resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/abs/2112.10752

    2022 arXiv

  50. [58]

    Automatic denoising of functional mri data: combining independent component analysis and hierarchical fusion of classifiers

    Gholamreza Salimi-Khorshidi, Gwena \"e lle Douaud, Christian F Beckmann, Matthew F Glasser, Ludovica Griffanti, and Stephen M Smith. Automatic denoising of functional mri data: combining independent component analysis and hierarchical fusion of classifiers. Neuroimage, 90: 0 4...

    2014

  51. [59]

    Reconstructing the mind's eye: fMRI -to-image with contrastive learning and diffusion priors

    Paul S Scotti, Atmadeep Banerjee, J Goode, Stepan Shabalin, A Nguyen, E Cohen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, K Norman, and T Abraham. Reconstructing the mind's eye: fMRI -to-image with contrastive learning and diffusion priors. Adv. Neural In...

    2023 arXiv

  52. [60]

    Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data

    Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. arXiv preprint a...

    2024 arXiv

  53. [61]

    Functional brain networks reflect spatial and temporal autocorrelation

    Maxwell Shinn, Amber Hu, Laurel Turner, Stephanie Noble, Katrin H Preller, Jie Lisa Ji, Flora Moujaes, Sophie Achard, Dustin Scheinost, R Todd Constable, John H Krystal, Franz X Vollenweider, Daeyeol Lee, Alan Anticevic, Edward T Bullmore, and John D Murray. Functional brain n...

    2023

  54. [62]

    Functional connectomics from resting-state fmri

    Stephen M Smith, Diego Vidaurre, Christian F Beckmann, Matthew F Glasser, Mark Jenkinson, Karla L Miller, Thomas E Nichols, Emma C Robinson, Gholamreza Salimi-Khorshidi, Mark W Woolrich, et al. Functional connectomics from resting-state fmri. Trends in cognitive sciences, 17 0...

    2013

  55. [63]

    Brain encoding models based on multimodal transformers can transfer across language and vision

    Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in Neural Information Processing Systems, 36, 2024

    2024

  56. [64]

    Self-supervised learning of brain dynamics from broad neuroimaging data

    Armin Thomas, Christopher R \'e , and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances in neural information processing systems, 35: 0 21255--21269, 2022

    2022

  57. [65]

    Aligning brain functions boosts the decoding of visual semantics in novel subjects

    Alexis Thual, Yohann Benchetrit, Felix Geilert, J \'e r \'e my Rapin, Iurii Makarov, Hubert Banville, and Jean-R \'e mi King. Aligning brain functions boosts the decoding of visual semantics in novel subjects. arXiv preprint arXiv:2312.06467, 2023

    2023 arXiv

  58. [66]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, 2022

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, 2022

    2022

  59. [67]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv \' e J \' e gou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012.12877, 2020. URL https://arxiv.org/abs/2012.12877

    2012 arXiv

  60. [68]

    Van Essen, Stephen M

    David C. Van Essen, Stephen M. Smith, Deanna M. Barch, Timothy E.J. Behrens, Essa Yacoub, and Kamil Ugurbil. The WU-Minn Human Connectome Project: An overview . NeuroImage, 80: 0 62--79, 10 2013. ISSN 10538119

    2013

  61. [69]

    Brain network dynamics are hierarchically organized in time

    Diego Vidaurre, Stephen M Smith, and Mark W Woolrich. Brain network dynamics are hierarchically organized in time. Proceedings of the National Academy of Sciences, 114 0 (48): 0 201705120, 2017

    2017

  62. [70]

    Videomae v2: Scaling video masked autoencoders with dual masking, 2023

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking, 2023

    2023

  63. [71]

    Neural encoding and decoding with deep learning for dynamic natural vision

    Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28 0 (12): 0 4136--4160, 2018

    2018

  64. [72]

    Relating transformers to models and neural representations of the hippocampal formation

    James C R Whittington, Joseph Warren, and Timothy E J Behrens. Relating transformers to models and neural representations of the hippocampal formation. December 2021

    2021

  65. [73]

    Using goal-driven deep learning models to understand sensory cortex

    Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19 0 (3): 0 356--365, 2016

    2016

  66. [74]

    Spherical demons: fast diffeomorphic landmark-free surface registration

    BT Thomas Yeo, Mert R Sabuncu, Tom Vercauteren, Nicholas Ayache, Bruce Fischl, and Polina Golland. Spherical demons: fast diffeomorphic landmark-free surface registration. IEEE transactions on medical imaging, 29 0 (3): 0 650--668, 2009

    2009

  67. [75]

    The organization of the human cerebral cortex estimated by intrinsic functional connectivity

    BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuncu, Danial Lashkari, Marisa Hollinshead, Joshua L Roffman, Jordan W Smoller, Lilla Z \"o llei, Jonathan R Polimeni, Bruce Fischl, Hesheng Liu, and Randy L Buckner. The organization of the human cerebral cortex estimat...

    2011

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.