REVIEW 3 major objections 5 minor 34 references
BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read BrainMT claims that combining a bidirectional Mamba scan over time with a global transformer over space lets fMRI models ingest 200 frames at once and predict sex and cognitive intelligence more accurately than existing voxel-based and…
desk verdict A promising empirical result on long fMRI sequences, but the architecture as described cannot be trained on the stated hardware—the paper is only publishable if the authors release code or correct the attention mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the temporal-first bidirectional Mamba block feeding a global transformer. In the Mamba block, a selective state-space model evolves a hidden state with input-dependent parameters, so a sequence of L = T × K tokens (200 volumes times roughly 220 spatial patches after downsampling) can be processed in linear time; arranging tokens with time as the leading dimension is what lets the scan capture long-range temporal correlations before spatial ones. The transformer block then applies Attention(Q, K, V) = Softmax(QK^T / $\sqrt$(d_head)) V over the same token sequence to model global spatial relationships, made tractable because the preceding convolution and Mamba stages reduce the sequence length. A learnable classification token X_cls is prepended, normalized, and passed through an MLP head for the final prediction.
What would settle it
Run the released implementation with the stated settings (T = 200, K roughly 220 spatial tokens, L roughly 44,000 tokens, 8 transformer heads, batch size 2 on a 48 GB GPU) and record peak GPU memory and the attention computation's input-output shapes; if the code cannot execute at that sequence length or the reported memory is exceeded, the central claim about a trainable global transformer over 200 frames is not supported as written.
Extended reading notes
Core claim
BrainMT's central claim is that long-range spatiotemporal dependencies in fMRI can be modeled end-to-end by splitting the work: a bidirectional selective state-space (Mamba) block handles the time dimension with linear cost, and a transformer block then applies global self-attention over the downsampled volume tokens to capture spatial relations. The paper reports state-of-the-art results on both tasks: for cognitive intelligence, MSE of 0.835 on the 1,075-subject cohort and 0.932 on the 6,000-subject cohort, with Pearson R of 0.41 and 0.24, compared with best-baseline MSEs of 0.914 and 0.994; for sex classification, accuracy of 96.28% on the 1,075-subject cohort and 97.91% on the 6,000-subject cohort, outperforming all baselines on the former and matching the leading prior method on the latter. The authors also claim this is the first hybrid Mamba-transformer approach with such temporal coverage in volumetric resting-state fMRI, that BrainMT is about 35.8% more memory-efficient than the leading 4D transformer with linear complexity in the number of time frames, and that Integrated Gradients maps implicate default-mode and frontoparietal regions for intelligence and temporal/frontal regions for sex.
Load-bearing premise
The result rests on the transformer actually computing one global attention step over all 200 time frames and all spatial patches at once inside a 48 GB GPU; if the implementation attends only within single volumes or after pooling, the described architecture is not what produced the numbers.
Editorial extensions
If this is right
- If the reported accuracies replicate, fMRI phenotype prediction no longer needs to truncate scans to 10–20 frames; full 200-frame volumes can be fed through a single forward pass.
- The reported memory saving and linear-in-time complexity mean larger cohorts, longer scans, or bigger batch sizes become feasible on a single 48 GB GPU, lowering the hardware bar for voxel-based neuroimaging models.
- The hybrid recipe—linear-time temporal scan plus global spatial attention—would likely transfer to other 4D medical images such as dynamic PET or contrast-enhanced MRI, where both the time course and the full volume matter.
- The Integrated Gradients maps give a concrete, model-derived list of brain regions that can be tested as biomarkers in independent datasets.
Reading between the lines
- The paper does not decompose its gains into the contribution of the temporal-first scan versus the transformer's global spatial attention; an ablation that isolates scan direction on the 1,075-subject cohort would pin down the mechanism behind the improvement.
- Because several correlation-based baselines sit at MSE near 1.0 on the 6,000-subject cohort, their errors are close to predicting the mean; reporting results on the original intelligence scale would show the real-world effect size behind the 6.23% reduction.
- An untested prediction is that temporal-first scanning helps most when slow hemodynamic dynamics carry signal; stratifying participants by scan length or low-frequency power and comparing BrainMT with and without the temporal-first scan would test this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BrainMT, a hybrid Mamba-transformer architecture for voxel-based fMRI phenotype prediction. The pipeline combines a 3D convolution block, a bidirectional Vision Mamba block with a temporal-first scan, and a multi-head self-attention transformer, followed by an MLP head on a classification token. The model is evaluated on sex classification and cognitive-intelligence regression using UK Biobank (6000 subjects) and HCP (1075 subjects), with repeated three-fold cross-validation against XGBoost, BrainNetCNN, BrainGNN, BrainNetTF, TFF, and SwiFT. The paper reports improved MSE/MAE/R and accuracy/AUROC relative to these baselines, plus ablations and Integrated Gradients maps. The central claim is state-of-the-art phenotype prediction while maintaining linear memory growth in the number of time frames.
Significance. If the empirical results hold, this is a potentially valuable contribution. The paper demonstrates that a 200-frame voxel-level input can be processed end-to-end, reports standard deviations from repeated cross-validation, includes a systematic ablation over components, layer counts, and input length, and links predictions to well-established brain networks through Integrated Gradients. The memory-efficiency comparison against SwiFT is also informative. However, the significance is conditional: the architecture as described is not computationally feasible for the stated sequence length, no code is provided to resolve the ambiguity, and the comparison with baselines may conflate architecture with input coverage. These issues prevent the current version from serving as a reproducible state-of-the-art claim.
major comments (3)
- [Section 2, 'Positional embedding' and 'Transformer block'; Section 3, 'Implementation details'; Fig. 1d] The manuscript defines Q, K, V in R^(L×Z) with L = T × K and sets T = 200 in Section 3, but it never reports H, W, D, or K. For typical HCP/UKB volumetric resolutions, the downsampled K is in the hundreds; for example, K = 220 gives L ≈ 44,000 tokens. Under the dense softmax attention in Eq. (5), this would produce roughly 1.9 × 10^9 attention entries per head, which is not trainable on a 48GB L40S GPU at batch size 2 without sequence-length reduction or sparse/windowed attention. The text does not mention FlashAttention, factorized attention, windowed attention, temporal pooling, or any other mechanism that would make this tractable. This also contradicts the claim in Section 3 and Fig. 1d that BrainMT 'maintains linear complexity in T', because attention over L = T × K is quadratic in T. The authors must either specify the actual sequence-reduction or attention mechanism, or correct the architecture description and the linear-complexity claim.
- [Abstract and Section 3, 'Implementation details'] The paper promises code at a GitHub URL, but no code, model definitions, or tensor-shape details are provided in the manuscript. Given the ambiguity about whether the transformer attends over the full L = T × K sequence or over a reduced sequence, the reader cannot determine which architecture was actually evaluated. The authors should provide at least a detailed tensor-shape table (T, H, W, D, K, L, Z after each block), a pseudocode description of the transformer block, and the exact number of attention heads; ideally, they should release the code or state precisely why the sequence length is computationally manageable.
- [Section 3, 'Experimental Setup and Results'] The paper states that baselines were run using their original implementations and hyperparameter settings, and the introduction notes that TFF and SwiFT process only 10–20 frames per forward pass, while BrainMT uses 200 frames. If the baseline results were produced with their native shorter windows, the comparison conflates architectural capacity with input temporal coverage. Please report the input length used for each baseline and include a matched-input-length comparison (e.g., SwiFT with 200 frames if feasible, or BrainMT with 20 frames) to isolate the architectural contribution to the reported gains.
minor comments (5)
- [Section 3, Table 3, Exp A] The text claims that 'T = 200 is optimal', but the table lists only T = 100 and T = 300; the T = 200 row is missing. Add the T = 200 result or revise the statement so it is directly supported by the ablation data.
- [Tables 1 and 2] The numeric cells are presented as concatenated values without visible plus-or-minus signs or color formatting, making it difficult to parse means and standard deviations. Please format each cell as 'mean ± std' and describe the color convention in the caption.
- [Fig. 1d and Section 3] The memory-efficiency comparison (35.8% more memory-efficient than SwiFT) lacks measurement details: batch size, sequence length, whether activations and gradients are included, and hardware are not specified. Add these details to the caption or the implementation section.
- [Section 2, 'Positional embedding' and 'Transformer block'] The text calls the transformer module a model of 'global spatial relationships', but the sequence is formed by concatenating spatial and temporal tokens into L = T × K. This terminology is misleading; the attention mechanism, as written, is spatiotemporal rather than purely spatial.
- [Table 3, Exp E] The ablation titled 'predicting functional connectivity correlations' introduces a new task without explaining the prediction target, the evaluation metric, or how SwiFT and BrainMT were adapted for it. Add a brief description of this experiment so the comparison is interpretable.
Circularity Check
No significant circularity: BrainMT's SOTA claims are external-benchmark comparisons; no prediction reduces to its inputs by construction.
full rationale
The central claim is empirical SOTA on UKB/HCP, established by comparing fixed architectures on held-out test splits. The paper's only fitted quantities are network weights and validation-chosen hyperparameters (T=200, layer counts); these are not relabeled predictions. Table 3 ablations select T=200 on validation, and the test metrics are then reported on a separate 15% test split; this is standard model selection, not a fitted-input-called-prediction. No equation in Section 2 defines the output metric in terms of the model's inputs or vice versa; self-attention, SSM discretization, and Mamba blocks are standard components with stated formulas, and the empirical claim does not rely on a theorem derived from those formulas. The self-citations (refs. [12], [21]) appear in background sentences and are not used to justify the architecture's performance or uniqueness. The GPU-memory/sequence-length concern raised by the skeptic is a reproducibility or architectural-consistency issue, not circularity: even if the described global attention over roughly 44,000 tokens is infeasible on a 48GB L40S, that does not make the reported MSE equal to a fitted parameter or to a self-citation. No uniqueness theorem is imported from the authors' prior work, and no known empirical pattern is renamed as a new result. Therefore the derivation chain is self-contained with respect to circularity.
Assumptions & free parameters
free parameters (4)
- Number of input time frames T =
200
- Mamba and transformer layer counts =
12 Mamba, 8 transformer
- SSM state dimension and expansion ratio =
state 16, expansion 2
- Patch size for volume partitioning =
H/4 x W/4 x D/4
assumptions (3)
- domain assumption The train/validation/test split is at the participant level, so frames from the same subject do not appear in both train and test.
- domain assumption Preprocessed fMRI volumes after global Z-score normalization are suitable direct inputs for a randomly initialized hybrid network.
- ad hoc to paper Default Mamba hyperparameters and the cosine learning rate schedule transfer to volumetric fMRI without task-specific tuning.
Cite this review
Pith. "Pith review of BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data." pith.science (2026). https://pith.science/paper/R23F22GB
@misc{pith2026250622591,
author = {Pith},
title = {Pith review of: BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/R23F22GB}},
note = {Machine review of arXiv:2506.22591}
}
read the original abstract
Recent advances in deep learning have made it possible to predict phenotypic measures directly from functional magnetic resonance imaging (fMRI) brain volumes, sparking significant interest in the neuroimaging community. However, existing approaches, primarily based on convolutional neural networks or transformer architectures, often struggle to model the complex relationships inherent in fMRI data, limited by their inability to capture long-range spatial and temporal dependencies. To overcome these shortcomings, we introduce BrainMT, a novel hybrid framework designed to efficiently learn and integrate long-range spatiotemporal attributes in fMRI data. Our framework operates in two stages: (1) a bidirectional Mamba block with a temporal-first scanning mechanism to capture global temporal interactions in a computationally efficient manner; and (2) a transformer block leveraging self-attention to model global spatial relationships across the deep features processed by the Mamba block. Extensive experiments on two large-scale public datasets, UKBioBank and the Human Connectome Project, demonstrate that BrainMT achieves state-of-the-art performance on both classification (sex prediction) and regression (cognitive intelligence prediction) tasks, outperforming existing methods by a significant margin. Our code and implementation details will be made publicly available at this https://github.com/arunkumar-kannan/BrainMT-fMRI
Figures
Reference graph
Works this paper leans on
-
[1]
Abraham, A., Milham, M.P., Di Martino, A., Craddock, R.C., Samaras, D., Thirion, B., Varoquaux, G.: Deriving reproducible biomarkers from multi-site resting-state data: An autism-based example. NeuroImage147, 736–745 (2017)
work page 2017
-
[2]
Neuroimage166, 400–424 (2018)
Alfaro-Almagro, F., Jenkinson, M., Bangerter, N.K., Andersson, J.L., Griffanti, L., Douaud, G., Sotiropoulos, S.N., Jbabdi, S., Hernandez-Fernandez, M., Vallee, E., et al.: Image processing and quality control for the first 10,000 brain imaging datasets from uk biobank. Neuroimage166, 400–424 (2018)
2018
-
[3]
In: Proceedings of the IEEE/CVF international confer- ence on computer vision
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF international confer- ence on computer vision. pp. 6836–6846 (2021)
2021
-
[4]
Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML. vol. 2, p. 4 (2021)
2021
-
[5]
In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining
Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. pp. 785–794 (2016)
2016
-
[6]
In: Medical Imaging with Deep Learning
Dsouza, N.S., Nebel, M.B., Crocetti, D., Robinson, J., Mostofsky, S., Venkatara- man, A.: M-gcn: A multimodal graph convolutional network to integrate functional and structural connectomics data to predict multidimensional phenotypic charac- terizations. In: Medical Imaging with Deep Learning. pp. 119–130. PMLR (2021)
work page 2021
-
[7]
Nature536(7615), 171–178 (2016)
Glasser, M.F., Coalson, T.S., Robinson, E.C., Hacker, C.D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C.F., Jenkinson, M., et al.: A multi- modal parcellation of human cerebral cortex. Nature536(7615), 171–178 (2016)
work page 2016
-
[8]
Neuroimage80, 105– 124 (2013) 10 A
Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Ander- sson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al.: The minimal preprocessing pipelines for the human connectome project. Neuroimage80, 105– 124 (2013) 10 A. Kannan et al
work page 2013
Show all 34 references
-
[9]
arXiv preprint arXiv:2312.00752 (2023)
Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)
2023 arXiv
-
[10]
arXiv preprint arXiv:2407.08083 (2024)
Hatamizadeh, A., Kautz, J.: Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083 (2024)
2024 arXiv
-
[11]
Advances in Neural Information Processing Systems 35, 25586–25599 (2022)
Kan, X., Dai, W., Cui, H., Zhang, Z., Guo, Y., Yang, C.: Brain network trans- former. Advances in Neural Information Processing Systems 35, 25586–25599 (2022)
2022
-
[12]
In: International Workshop on Machine Learn- ing in Clinical Neuroimaging
Kannan, A., Caffo, B., Venkataraman, A.: Gaming the brain: Investigating the cross-modal relationships between functional connectivity and structural features using generalized additive models. In: International Workshop on Machine Learn- ing in Clinical Neuroimaging. pp. 166–...
2025
-
[13]
NeuroImage 146, 1038– 1049 (2017)
Kawahara, J., Brown, C.J., Miller, S.P., Booth, B.G., Chau, V., Grunau, R.E., Zwicker, J.G., Hamarneh, G.: Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment. NeuroImage 146, 1038– 1049 (2017)
2017
-
[14]
Neu- roImage 199, 651–662 (2019)
Khosla, M., Jamison, K., Kuceyeski, A., Sabuncu, M.R.: Ensemble learning with 3d convolutional neural networks for functional connectome-based prediction. Neu- roImage 199, 651–662 (2019)
2019
-
[15]
Magnetic resonance imaging64, 101–121 (2019)
Khosla, M., Jamison, K., Ngo, G.H., Kuceyeski, A., Sabuncu, M.R.: Machine learn- ing in resting-state fmri analysis. Magnetic resonance imaging64, 101–121 (2019)
2019
-
[16]
Advances in Neural Information Processing Systems 36, 42015–42037 (2023)
Kim, P., Kwon, J., Joo, S., Bae, S., Lee, D., Jung, Y., Yoo, S., Cha, J., Moon, T.: Swift: Swin 4d fmri transformer. Advances in Neural Information Processing Systems 36, 42015–42037 (2023)
2023
-
[17]
In: European Conference on Computer Vision
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., Qiao, Y.: Videomamba: State space model for efficient video understanding. In: European Conference on Computer Vision. pp. 237–255. Springer (2024)
2024
-
[18]
Neurocomputing388, 280–287 (2020)
Li, W., Lin, X., Chen, X.: Detecting alzheimer’s disease based on 4d fmri: An exploration under deep learning framework. Neurocomputing388, 280–287 (2020)
2020
-
[19]
Medical Image Analysis74, 102233 (2021)
Li, X., Zhou, Y., Dvornek, N., Zhang, M., Gao, S., Zhuang, J., Scheinost, D., Staib, L.H., Ventola, P., Duncan, J.S.: Braingnn: Interpretable brain graph neural network for fmri analysis. Medical Image Analysis74, 102233 (2021)
2021
-
[20]
Statistical science (4), 439– 464 (2008)
Lindquist, M.A.: The statistical analysis of fmri data. Statistical science (4), 439– 464 (2008)
2008
-
[21]
Annual Review of Statistics and Its Application12 (2024)
Lindquist, M.A., Smith, B.B., Kannan, A., Zhao, A., Caffo, B.: Measuring the func- tioning human brain. Annual Review of Statistics and Its Application12 (2024)
2024
-
[22]
In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https://openreview.net/forum? id=ZgtLQQR1K7
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., Jiao, J., Liu, Y.: VMamba: Visual state space model. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https://openreview.net/forum? id=ZgtLQQR1K7
2024
-
[23]
In: International Conference on Medical Imaging with Deep Learning
Malkiel, I., Rosenman, G., Wolf, L., Hendler, T.: Self-supervised transformers for fmri representation. In: International Conference on Medical Imaging with Deep Learning. pp. 895–913. PMLR (2022)
2022
-
[24]
Trends in cognitive sciences15(10), 483–506 (2011)
Menon, V.: Large-scale brain networks and psychopathology: a unifying triple net- work model. Trends in cognitive sciences15(10), 483–506 (2011)
2011
-
[25]
In: Machine Learning for Health
Nguyen, S., Ng, B., Kaplan, A.D., Ray, P.: Attend and decode: 4d fmri task state decoding using attention models. In: Machine Learning for Health. pp. 267–279. PMLR (2020)
2020
-
[26]
In: European Conference on Computer Vision
Park, J., Kim, H.S., Ko, K., Kim, M., Kim, C.: Videomamba: Spatio-temporal selective state space model. In: European Conference on Computer Vision. pp. 1–18. Springer (2024) BrainMT 11
2024
-
[27]
Proceedings of the National Academy of Sciences 121(9), e2310012121 (2024)
Ryali, S., Zhang, Y., de Los Angeles, C., Supekar, K., Menon, V.: Deep learning models reveal replicable, generalizable, and behaviorally relevant sex differences in human functional brain organization. Proceedings of the National Academy of Sciences 121(9), e2310012121 (2024)
2024
-
[28]
Journal of Neuroscience31(12), 4407–4420 (2011)
Sestieri, C., Corbetta, M., Romani, G.L., Shulman, G.L.: Episodic memory re- trieval, parietal cortex, and the default mode network: functional and topographic analyses. Journal of Neuroscience31(12), 4407–4420 (2011)
2011
-
[29]
Frontiers in Neuroimaging2, 1178359 (2023)
Smith, B.B., Zhao, Y., Lindquist, M.A., Caffo, B.: Regression models for partially localized fmri connectivity analyses. Frontiers in Neuroimaging2, 1178359 (2023)
2023
-
[30]
Neuroimage80, 144–168 (2013)
Smith, S.M., Beckmann, C.F., Andersson, J., Auerbach, E.J., Bijsterbosch, J., Douaud, G., Duff, E., Feinberg, D.A., Griffanti, L., Harms, M.P., et al.: Resting- state fmri in the human connectome project. Neuroimage80, 144–168 (2013)
2013
-
[31]
PLoS medicine 12(3), e1001779 (2015)
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al.: Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine 12(3), e100...
2015
-
[32]
In: International conference on machine learning
Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International conference on machine learning. pp. 3319–3328. PMLR (2017)
2017
-
[33]
Cerebral cortex30(2), 824–835 (2020)
Weis, S., Patil, K.R., Hoffstaedter, F., Nostro, A., Yeo, B.T., Eickhoff, S.B.: Sex classification by resting state brain connectivity. Cerebral cortex30(2), 824–835 (2020)
2020
-
[34]
arXiv preprint arXiv:2401.09417 (2024)
Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., Wang, X.: Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 (2024)
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.