COBRA: A Continual Learning Approach to Vision-Brain Understanding

Arabinda Kumar Choudhary; Khoa Luu; Manuel Serna-Aguilera; Pawan Sinha; Xin Li; Xuan-Bac Nguyen

REVIEW 4 major objections 5 minor 1 cited by

COBRA: A Continual Learning Approach to Vision-Brain Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read COBRA claims that continually adding new fMRI subjects to a vision-brain decoder does not have to erase knowledge of previous subjects, and that a modular design achieves state-of-the-art reconstruction and captioning without storing…

desk verdict A genuinely new modular continual-learning framework for fMRI-to-CLIP alignment with strong NSD experiments, but the reconstruction claim is untestable as written because no image decoder is specified. read the letter →

arxiv 2411.17475 v3 pith:IPJLUACU submitted 2024-11-25 cs.CV

Xuan-Bac Nguyen , Manuel Serna-Aguilera , Arabinda Kumar Choudhary , Pawan Sinha , Xin Li , Khoa Luu This is my paper · ORCID

classification cs.CV

keywords continuallearningvision-brainunderstandingfMRIdecodingimagereconstructionfrombrainactivitycaptioningprompt-basedcatastrophicforgettingNaturalScenesDataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces COBRA, a continual-learning framework for vision-brain understanding: reconstructing or captioning what a person saw from their fMRI activity when new participants are added over time. Existing per-subject decoders ignore patterns shared across people, while unified decoders suffer catastrophic forgetting when adapted to new subjects. COBRA separates the problem into a frozen shared module that captures brain-activation patterns common to all subjects and per-subject prompt and transformer modules that are trained only for the new person. The paper argues that this removes the need to retain prior subjects' data, prevents forgetting, and gives state-of-the-art image reconstruction and captioning results on the Natural Scenes Dataset.

What carries the argument

The key machinery is the three-module split with a frozen common encoder. The SC module maps 2D-flattened fMRI patches into a CLS token plus patch tokens and predicts object presence; the PSS module treats subject identity as a prompt by selecting the 30 tokens most similar to the CLS token; the MRIFormer concatenates common and specific tokens and uses a transformer decoder, rather than a linear layer, to translate them into CLIP feature length. The load-bearing design choice is that when a new subject arrives only the new PSS and MRIFormer modules are updated, so no gradient step can overwrite the knowledge stored in the shared module.

What would settle it

Compute the round-trip error of the flattening: take a subject's 3D fMRI volume, project it to the 2D fsaverage grid, invert the projection, and compare voxel by voxel with the original. If the round trip is not identity, the paper's claim of a bijective, no-information-loss projection is false; a complementary check is to train COBRA directly on 3D patch inputs and see whether the reported SSIM and CLIP margins persist.

Watch

Extended reading notes

Core claim

COBRA's central claim is that continual vision-brain understanding can be solved by decomposing fMRI decoding into a commonality branch and a specificity branch. A Subject Commonality module, a vision transformer supervised by binary cross-entropy over 80 COCO object classes, learns shared visuo-brain patterns from the subjects seen so far and is then frozen. For each new subject, a Prompt-based Subject Specific module selects the 30 most relevant tokens using a key-query mechanism, and the MRIFormer encoder-decoder translates the combined common and specific features into CLIP space, where contrastive loss aligns them with the visual stimulus. Because only the new subject's PSS and MRIFormer parameters are trained, earlier modules are untouched, so knowledge from prior subjects is retained without rehearsal. The paper reports that this design reaches SSIM around 0.32-0.33 and CLIP scores around 92-97% in two rehearsal-free continual setups, outperforming LwF and PLOP, and also reports state-of-the-art reconstruction and captioning results when trained on all subjects.

Load-bearing premise

The whole pipeline rests on the assumption that flattening the brain's curved cortical surface into a 2D $224\times224$ image loses no information and preserves spatial relationships; if that projection distorts or drops data, every module downstream is working on an altered signal.

Editorial extensions

If this is right

New fMRI subjects could be added to a decoder without access to previous participants' recordings, which matters when brain data are too privacy-sensitive to store or share.
Per-subject training cost drops to training a small prompt module and transformer for roughly two hours per subject, instead of retraining a full model on all subjects.
If the shared module genuinely captures common brain patterns, performance should improve as more subjects contribute to it, rather than degrade with each new arrival.
Because MRIFormer maps into a shared CLIP space, the same continual scheme could extend to other decoding targets such as captions, speech, or visual-semantic queries.
The reported gains over prior continual baselines suggest that catastrophic forgetting in fMRI decoding is not inevitable and can be engineered away by modularity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the reported advantage could partly come from the 2D cortical-flattening representation itself rather than from the continual-learning modules; training the same modules on 3D patch inputs would isolate that contribution.
Beyond the paper: the SC module's object-prediction supervision is tied to 80 COCO classes, so on datasets with different stimulus categories the shared representation may need re-supervision, a limitation the paper only partially acknowledges.
Beyond the paper: the prompt-selection mechanism resembles retrieval-based continual learning, and a natural test is whether it generalizes to subjects with atypical cortical anatomy, where the fsaverage template fits poorly.
Beyond the paper: since the paper shows COBRA can be combined with PLOP for further gains, a follow-up could probe whether rehearsal-free performance saturates at some number of subjects and whether synthetic prompt replay would push it further.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: COBRA is a new and reasonably well-engineered combination of parameter-isolation continual learning and fMRI-to-CLIP alignment, with thorough experiments on NSD. But as written, the reconstruction claims are not reproducible because the method section stops at CLIP-space alignment and never says how the CLIP embedding becomes an image.

The core idea is sensible: freeze a commonality transformer trained across subjects to predict COCO object labels, add per-subject prompt selection, and use a transformer decoder to map into CLIP space. The paper deserves credit for real evidence: the SC module transfers to unseen subjects with F1=0.84, the ablations on prompt length and the regularization loss are informative, and the model-size comparison is useful. The continual-learning setup is honest in that old modules are frozen, so the absence of forgetting is by construction, but that is a legitimate design choice.

Three soft spots, in increasing order of severity. First, the captioning SOTA is slightly overstated: Table 15 shows SPICE below UMBRAE (12.6 vs 12.79), and Table 14 has them second on SSIM. The abstract's "state-of-the-art" should be qualified. Second, the 2D flattening claim is unproven. Section 3.2 asserts a bijective, information-preserving projection that maintains geodesic distance, but the appendix only illustrates 1D vs 2D sparsity; it does not demonstrate isometry. Flattening a curved cortical surface to a 224x224 grid is not generally lossless, and the claim should be downgraded to an approximation. Third, and most load-bearing: no image decoder is ever described. Section 3.5 and Algorithm 1 output a CLIP-space feature, yet Tables 2 and 14 report SSIM and PixCorr. Unless the authors identify the pretrained diffusion prior or generator used to convert the CLIP embedding into pixels, and control for it across baselines, the reconstruction numbers are not attributable to COBRA. This is a reproducibility gap, not necessarily a fatal flaw in the architecture.

Who this is for: researchers in cross-subject fMRI decoding and continual learning will find the modular design worth studying. I would send this to peer review, but with a required major revision: specify the full reconstruction pipeline, provide code or detailed hyperparameters, and soften the claims to match the tables. The framework is new and the experiments are extensive; it deserves a serious referee.

Recommendation: engage with it, but insist on clarification of the image decoder before citing the reconstruction results.

Referee Report

4 major / 5 minor

Summary. The paper introduces COBRA, a continual learning framework for vision-brain understanding (VBU) that processes fMRI signals projected onto a 2D fsaverage template. COBRA combines a frozen shared Subject Commonality (SC) module, a Prompt-based Subject Specific (PSS) module that selects top-k tokens per subject, and an MRIFormer transformer encoder-decoder that aligns fMRI features with CLIP image embeddings via a contrastive loss. In the continual setup, new subjects are accommodated by training new PSS and MRIFormer modules, leaving previous subjects' modules untouched. Experiments on the Natural Scenes Dataset are reported for two continual learning protocols, covering reconstruction metrics such as SSIM and CLIP score, captioning metrics across ten standard measures, ablations on prompt length and loss weights, and comparisons with prior VBU and continual learning methods.

Significance. If the empirical results hold, the modular architecture is a plausible and practical way to add new fMRI subjects without storing past data, and the explicit evaluation of subject-commonality generalization (F1=0.84 on unseen subjects) and subject-specific prompt selection (98.5% accuracy) are useful contributions. The paper includes extensive experiments with error bars, multiple continual learning scenarios, and a fairly complete set of ablations. The main limitation is that the reconstruction pipeline is not described, which makes the central reconstruction SOTA claim unverifiable as written.

major comments (4)

[Section 3.5, Tables 2 and 14, Figures 4-5] The reconstruction results are not attributable to COBRA as written. The method section stops at producing fMRI-aligned CLIP features via the contrastive loss in Eq. (9), and no diffusion prior, image decoder, or generator is specified anywhere in the manuscript. Yet Tables 2 and 14 and Figures 4-5 report pixel-level and image-level metrics (SSIM, PixCorr, AlexNet/Inception/CLIP similarities) and show reconstructed images. The paper must specify the full image decoding pipeline, state whether any pretrained decoder is used, and control for that decoder in comparisons with prior methods. Without this, the reconstruction SOTA claim is untestable.
[Section 3.2 and Appendix 7.2] The claim that the 3D-to-2D fsaverage flattening is a bijective projection with 'no information loss' and that it 'preserves geodesic distance and spatial relationships' is unsupported and, for a folded cortical surface, implausible as stated. A plane cannot be globally isometric to a curved surface; flattening necessarily introduces distortions or cuts unless the surface is developable. Appendix 7.2 only illustrates that 1D contiguous samples become scattered in 2D; it does not prove information preservation. Since all modules consume 224x224 2D patches, this assertion is load-bearing. Please revise to describe the specific projection used, cite authoritative references for its properties, and replace 'no information loss' with a qualified statement about the approximation.
[Section 3.6 and Tables 2-8] The 'no catastrophic forgetting' result is largely by construction under parameter isolation: previous PSS and MRIFormer modules are frozen, so old subjects' features cannot be overwritten. The continual learning comparison against LwF and PLOP, which update a shared network, is therefore not an apples-to-apples comparison. To support the claim of state-of-the-art continual learning performance, the authors should compare COBRA with architecture-based continual learning methods (e.g., progressive neural networks, PackNet, or prompt-based CL baselines) or clearly qualify that the comparison is limited to rehearsal-free regularization/distillation baselines.
[Section 3.6 and Algorithm 1] It is unclear whether the SC module is frozen after the first training step or continues to be updated when new subjects are added. The text says 'we do not retrain or update the entire model' and only new PSS and MRIFormer modules are created, yet Algorithm 1 includes the SC loss Lc in the total loss at every step, and feeding new subjects' data through the SC module could involve gradient updates. Please specify the exact update schedule for the SC module across training steps and clarify how the commonality loss is used after the initial stage.

minor comments (5)

[Eq. (11) vs Algorithm 1] The loss in Eq. (11) is written as λsc Lcon, but Algorithm 1 line 18 uses 'LsoftCLIP' for the same term; please unify the notation.
[Section 3.4] The statement that the PSS module training is 'label-free' is misleading because the subject identity is used as a supervised label; this is a self-supervised pretext task, not label-free learning.
[Table 14] The results in Table 14 are reported without error bars or the number of repeat evaluations, unlike the continual learning tables; please add error bars or state the evaluation protocol.
[Section 4.7 and Table 17] The ablation labeled 'Effectiveness of the vision-brain data representation' varies both the input signal (partial ROI vs full signals) and the fMRI-to-CLIP transformation (linear vs MRIFormer) simultaneously, so the individual contributions of these two factors cannot be separated; please present a two-factor ablation.
[Section 7.1] There is a typo: 'floc−face and floc−face' should likely be 'floc−face and floc−body'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: COBRA's learning targets are external labels and embeddings, and its forgetting-free behavior follows from parameter isolation.

full rationale

The COBRA pipeline does not contain a circular derivation. The SC module is trained with an external object-label BCE loss (Eq. 3); the PSS module is trained with subject-identity classification (Eq. 5) whose labels come from dataset bookkeeping; and MRIFormer is aligned to CLIP embeddings of the actual stimulus (Eq. 9). None of these targets is an output of COBRA itself, so the learned features are not self-predictions. The continual-learning claim ('leaves previous subjects' PSS and MRIFormer unaffected') is an architectural consequence of freezing old modules in Section 3.6, not a fitted parameter renamed as a result. The extensive self-citations (e.g., refs [43]-[56]) occur in related-work enumerations and are not load-bearing; no uniqueness theorem or prior ansatz is imported to force the design. The paper does contain two notable non-circularity concerns: the Section 3.2 claim that fsaverage 2D flattening is a lossless bijection is asserted without proof, and Section 3.5 stops at CLIP-space alignment with no image decoder specified before Tables 2/14 report pixel-level metrics. These are completeness and correctness issues, not cases where an output is equivalent to an input by construction. The central claims are externally benchmarked on NSD and the SC module is tested on unseen subjects (F1=0.84), so the derivation is self-contained.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or scientific entities. SC, PSS, and MRIFormer are neural network components, not postulated phenomena, so the invented-entities ledger is empty.

free parameters (3)

top-k prompt length = 30
Selected by ablation in Table 18; controls the number of subject-specific prompt tokens in the PSS module.
center margin in regularization loss = not reported
Appears in Eqn (10) as the minimum separation between subject center vectors, but no value or tuning procedure is reported.
loss weights lambda_c, lambda_s, lambda_sc, lambda_reg = all set to 1.0
Set by hand; the ablation only varies lambda_reg and lambda_con, so the reported performance depends on this choice.

assumptions (4)

ad hoc to paper The 2D flattening of the 3D fsaverage cortical surface is bijective and preserves all spatial information.
Section 3.2 asserts this without proof or reference; flattening a curved surface to a plane generally introduces distortion, so this is a strong unverified premise.
domain assumption There exist common vision-brain activation patterns across subjects that a single transformer can learn from 2D fMRI, and these patterns transfer to unseen subjects.
Motivated by neuroscience references [12-21] and tested via F1=0.84 in Section 4.7, but the transfer threshold is not theoretically grounded.
domain assumption Subject identity classification through top-k selected prompt tokens yields features specific enough to improve CLIP-space alignment for reconstruction.
The PSS module is trained with a subject-identity loss in Eqn (5); the paper does not show that the selected prompts, rather than the auxiliary task itself, cause the reconstruction gains.
domain assumption CLIP embeddings are an adequate shared target space for fMRI decoding and captioning.
Used by prior vision-brain works; COBRA relies on it for the contrastive loss and decoder translation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of COBRA: A Continual Learning Approach to Vision-Brain Understanding." pith.science (2026). https://pith.science/paper/IPJLUACU

@misc{pith2026241117475,
  author       = {Pith},
  title        = {Pith review of: COBRA: A Continual Learning Approach to Vision-Brain Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPJLUACU}},
  note         = {Machine review of arXiv:2411.17475}
}

read the original abstract

Vision-Brain Understanding (VBU) aims to extract visual information perceived by humans from brain activity recorded through functional Magnetic Resonance Imaging (fMRI). Despite notable advancements in recent years, existing studies in VBU continue to face the challenge of catastrophic forgetting, where models lose knowledge from prior subjects as they adapt to new ones. Addressing continual learning in this field is, therefore, essential. This paper introduces a novel framework called Continual Learning for Vision-Brain (COBRA) to address continual learning in VBU. Our approach includes three novel modules: a Subject Commonality (SC) module, a Prompt-based Subject Specific (PSS) module, and a transformer-based module for fMRI, denoted as MRIFormer module. The SC module captures shared vision-brain patterns across subjects, preserving this knowledge as the model encounters new subjects, thereby reducing the impact of catastrophic forgetting. On the other hand, the PSS module learns unique vision-brain patterns specific to each subject. Finally, the MRIFormer module contains a transformer encoder and decoder that learns the fMRI features for VBU from common and specific patterns. In a continual learning setup, COBRA is trained in new PSS and MRIFormer modules for new subjects, leaving the modules of previous subjects unaffected. As a result, COBRA effectively addresses catastrophic forgetting and achieves state-of-the-art performance in both continual learning and vision-brain reconstruction tasks, surpassing previous methods.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understanding
cs.CV 2024-11 conditional novelty 4.0 of 10

A quantum-inspired quadratic connectivity layer for fMRI voxels reaches top-1 image retrieval of 95.5% and brain retrieval of 95.3% on NSD, ahead of prior MindEye results.

Reference graph

Works this paper leans on

90 extracted references · 30 canonical work pages · cited by 1 Pith paper

[1]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Wang, S., Liu, S., Tan, Z., Wang, X.: Mindbridge: A cross-subject brain decoding framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11333–11342 (2024)

2024
[2]

In: 2022 International Joint Conference on Neural Networks (IJCNN), pp

Ozcelik, F., Choksi, B., Mozafari, M., Reddy, L., VanRullen, R.: Reconstruc- tion of perceived images from fmri patterns and semantic brain exploration using instance-conditioned gans. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2022). IEEE

2022
[3]

PLoS computational biology 15(1), 1006633 (2019) 28

Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS computational biology 15(1), 1006633 (2019) 28

2019
[5]

Advances in Neural Information Processing Systems 36 (2024)

Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., Ver- linde, N., Yundler, E., Weisberg, D., Norman, K., et al.: Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems 36 (2024)

2024
[6]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Quan, R., Wang, W., Tian, Z., Ma, F., Yang, Y.: Psychometry: An omnifit model for image reconstruction from human brain activity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 233–243 (2024)

2024
[7]

arXiv preprint arXiv:2403.18211 (2024)

Huo, J., Wang, Y., Qian, X., Wang, Y., Li, C., Feng, J., Fu, Y.: Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi- level modulation. arXiv preprint arXiv:2403.18211 (2024)

arXiv 2024
[8]

arXiv preprint arXiv:2403.11207 (2024)

Scotti, P.S., Tripathy, M., Villanueva, C.K.T., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K.A., et al.: Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403.11207 (2024)

arXiv 2024
[9]

In: Learning to Learn, pp

Thrun, S.: Lifelong learning algorithms. In: Learning to Learn, pp. 181–209. Springer, ??? (1998)

1998

Show all 90 references

[10]

Trends in cognitive sciences 3(4), 128–135 (1999)

French, R.M.: Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3(4), 128–135 (1999)

1999
[11]

Connection Science 7(2), 123–146 (1995)

Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science 7(2), 123–146 (1995)

1995
[12]

Science 293(5539), 2470–2473 (2001)

Downing, P.E., Jiang, Y., Shuman, M., Kanwisher, N.: A cortical area selective for visual processing of the human body. Science 293(5539), 2470–2473 (2001)

2001
[13]

Neuropsychologia 49(5), 821–829 (2011)

Costantini, M., Urgesi, C., Galati, G., Romani, G.L., Aglioti, S.M.: Haptic per- ception and body representation in lateral and medial occipito-temporal cortices. Neuropsychologia 49(5), 821–829 (2011)

2011
[14]

Nature 392(6676), 598–601 (1998)

Epstein, R., Kanwisher, N.: A cortical representation of the local visual environ- ment. Nature 392(6676), 598–601 (1998)

1998
[15]

Cerebral cortex 27(1), 146–161 (2017) 29

Weiner, K.S., Barnett, M.A., Lorenz, S., Caspers, J., Stigliani, A., Amunts, K., Zilles, K., Fischl, B., Grill-Spector, K.: The cytoarchitecture of domain-specific regions in human high-level visual cortex. Cerebral cortex 27(1), 146–161 (2017) 29

2017
[16]

Epstein, R., Harris, A., Stanley, D., Kanwisher, N.: The parahippocampal place area: recognition, navigation, or encoding? Neuron 23(1), 115–125 (1999)

1999
[17]

Annual review of vision science 5(1), 373–397 (2019)

Epstein, R.A., Baker, C.I.: Scene perception in the human brain. Annual review of vision science 5(1), 373–397 (2019)

2019
[18]

Annual review of vision science 1(1), 393–416 (2015)

Duchaine, B., Yovel, G.: A revised neural framework for face processing. Annual review of vision science 1(1), 393–416 (2015)

2015
[19]

Proceedings of the National Academy of Sciences 93(2), 922–927 (1996)

Haxby, J.V., Ungerleider, L.G., Horwitz, B., Maisog, J.M., Rapoport, S.I., Grady, C.L.: Face encoding and recognition in the human brain. Proceedings of the National Academy of Sciences 93(2), 922–927 (1996)

1996
[20]

Human Brain Mapping 3(2), 68–82 (1995)

Haxby, J.V., Ungerleider, L.G., Horwitz, B., Rapoport, S.I., Grady, C.L.: Hemi- spheric differences in neural systems for face working memory: A pet-rcbf study. Human Brain Mapping 3(2), 68–82 (1995)

1995
[21]

Annual review of vision science 3(1), 167–196 (2017)

Grill-Spector, K., Weiner, K.S., Kay, K., Gomez, J.: The functional neuroanatomy of human face perception. Annual review of vision science 3(1), 167–196 (2017)

2017
[22]

arXiv preprint arXiv:2411.13378 (2024)

Nguyen, H.-Q., Nguyen, X.-B., Churchill, H., Choudhary, A.K., Sinha, P., Khan, S.U., Luu, K.: Quantum-brain: Quantum-inspired neural network approach to vision-brain understanding. arXiv preprint arXiv:2411.13378 (2024)

2024 arXiv
[23]

Quantum Machine Intelligence 6(2), 61 (2024)

Nguyen, X.-B., Nguyen, H.-Q., Churchill, H., Khan, S.U., Luu, K.: Quantum visual feature encoding revisited. Quantum Machine Intelligence 6(2), 61 (2024)

2024
[24]

International Journal of Computer Vision, 1–26 (2025)

Truong, T.-D., Nguyen, H.-Q., Nguyen, X.-B., Dowling, A., Li, X., Luu, K.: Insect- foundation: A foundation model and large multimodal dataset for vision-language insect understanding. International Journal of Computer Vision, 1–26 (2025)

2025
[25]

ACM Computing Surveys 55(9), 1–35 (2023)

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55(9), 1–35 (2023)

2023
[26]

https://arxiv.org/abs/2404.07202

Xia, W., Charette, R., ¨Oztireli, C., Xue, J.-H.: UMBRAE: Unified Multimodal Brain Decoding (2024). https://arxiv.org/abs/2404.07202

2024 arXiv
[27]

CoRR abs/1503.03585 (2015) 1503.03585

Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsuper- vised learning using nonequilibrium thermodynamics. CoRR abs/1503.03585 (2015) 1503.03585

2015 arXiv
[28]

arXiv preprint arxiv:2006.11239 (2020)

Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.11239 (2020)

2020 arXiv
[29]

CoRR abs/2105.05233 (2021) 2105.05233 30

Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. CoRR abs/2105.05233 (2021) 2105.05233 30

2021 arXiv
[30]

https://arxiv.org/abs/ 2112.10752

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models (2022). https://arxiv.org/abs/ 2112.10752

2022 arXiv
[31]

https://arxiv.org/abs/2211

Xu, X., Wang, Z., Zhang, E., Wang, K., Shi, H.: Versatile Diffusion: Text, Images and Variations All in One Diffusion Model (2024). https://arxiv.org/abs/2211. 08332

2024
[32]

https: //arxiv.org/abs/2112.08654

Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to Prompt for Continual Learning (2022). https: //arxiv.org/abs/2112.08654

2022 arXiv
[33]

https://arxiv.org/abs/1809.05922

Hayes, T.L., Cahill, N.D., Kanan, C.: Memory Efficient Experience Replay for Streaming Learning (2019). https://arxiv.org/abs/1809.05922

2019 arXiv
[34]

https://arxiv.org/abs/1902.10486

Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P.K., Torr, P.H.S., Ranzato, M.: On Tiny Episodic Memories in Continual Learning (2019). https://arxiv.org/abs/1902.10486

2019 arXiv
[35]

https://arxiv.org/abs/ 2002.08165

Chaudhry, A., Gordo, A., Dokania, P.K., Torr, P., Lopez-Paz, D.: Using Hindsight to Anchor Past Knowledge in Continual Learning (2021). https://arxiv.org/abs/ 2002.08165

2021 arXiv
[36]

https://arxiv.org/abs/1812.00420

Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient Lifelong Learning with A-GEM (2019). https://arxiv.org/abs/1812.00420

2019 arXiv
[37]

https://arxiv

Buzzega, P., Boschini, M., Porrello, A., Abati, D., Calderara, S.: Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020). https://arxiv. org/abs/2004.07211

2020 arXiv
[38]

https://arxiv.org/abs/1611.07725

Rebuffi, S.-A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: Incremental Classifier and Representation Learning (2017). https://arxiv.org/abs/1611.07725

2017 arXiv
[39]

In: 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp

Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 909–910 (2015). https://doi.org/10.1109/ALLERTON.2015.7447103

2015
[40]

https://arxiv.org/abs/2110.00175

Pham, Q., Liu, C., Hoi, S.: DualNet: Continual Learning, Fast and Slow (2021). https://arxiv.org/abs/2110.00175

2021 arXiv
[41]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

Cha, H., Lee, J., Shin, J.: Co2l: Contrastive continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9516–9525 (2021)

2021
[42]

https://arxiv.org/abs/1905.13260

Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large Scale Incremental Learning (2019). https://arxiv.org/abs/1905.13260

2019 arXiv
[43]

In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp

Nguyen, X.-B., Lee, G.-S., Kim, S.-H., Yang, H.-J.: Audio-video based emotion 31 recognition using minimum cost flow algorithm. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3737–3741 (2019). IEEE

2019
[44]

International Journal of Contents15(4), 8–15 (2019)

Nguyen-Xuan, B., Lee, G.-S.: Sketch recognition using lstm with attention mech- anism and minimum cost flow algorithm. International Journal of Contents15(4), 8–15 (2019)

2019
[45]

IEEE Access 8, 162973–162981 (2020)

Nguyen, X.-B., Lee, G.S., Kim, S.H., Yang, H.J.: Self-supervised learning based on spatial awareness for medical image analysis. IEEE Access 8, 162973–162981 (2020)

2020
[46]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Nguyen, X.-B., Bui, D.T., Duong, C.N., Bui, T.D., Luu, K.: Clusformer: A trans- former based clustering approach to unsupervised large-scale face and visual landmark recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10847–1...

2021
[47]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Nguyen, P., Quach, K.G., Duong, C.N., Le, N., Nguyen, X.-B., Luu, K.: Multi- camera multiple 3d object tracking on the move for autonomous vehicles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2569–2578 (2022)

2022
[48]

arXiv preprint arXiv:2205.15948 (2022)

Nguyen, X.B., Bisht, A., Churchill, H., Luu, K.: Two-dimensional quantum mate- rial identification via self-attention and soft-labeling in deep learning. arXiv preprint arXiv:2205.15948 (2022)

2022 arXiv
[49]

arXiv preprint arXiv:2308.00262 (2023)

Nguyen, X.-B., Liu, X., Li, X., Luu, K.: The algonauts project 2023 challenge: Uark-ualbany team solution. arXiv preprint arXiv:2308.00262 (2023)

2023 arXiv
[50]

arXiv preprint arXiv:2312.00236 (2023)

Nguyen, X.-B., Li, X., Khan, S.U., Luu, K.: Brainformer: Modeling mri brain functions to machine vision. arXiv preprint arXiv:2312.00236 (2023)

2023 arXiv
[51]

arXiv preprint arXiv:2304.07408 (2023)

Nguyen, X.-B., Duong, C.N., Savvides, M., Roy, K., Churchill, H., Luu, K.: Fair- ness in visual clustering: A novel transformer clustering approach. arXiv preprint arXiv:2304.07408 (2023)

2023 arXiv
[52]

arXiv preprint arXiv:2405.18808 (2024)

Nguyen, X.-B., Jang, H., Li, X., Khan, S.U., Sinha, P., Luu, K.: Bractive: A brain activation approach to human visual brain learning. arXiv preprint arXiv:2405.18808 (2024)

2024 arXiv
[53]

arXiv preprint arXiv:2406.00843 (2024)

Nguyen, H.-Q., Nguyen, X.B., Chen, S.Y.-C., Churchill, H., Borys, N., Khan, S.U., Luu, K.: Diffusion-inspired quantum noise mitigation in parameterized quantum circuits. arXiv preprint arXiv:2406.00843 (2024)

2024 arXiv
[54]

arXiv preprint arXiv:2408.03596 (2024) 32

Nguyen, X.-B., Nguyen, H.-Q., Churchill, H., Khan, S.U., Luu, K.: Hierarchi- cal quantum control gates for functional mri understanding. arXiv preprint arXiv:2408.03596 (2024) 32

2024 arXiv
[55]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Nguyen, H.-Q., Truong, T.-D., Nguyen, X.B., Dowling, A., Li, X., Luu, K.: Insect-foundation: A foundation model and large-scale 1m dataset for visual insect understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21945–21955 (2024)

2024
[56]

arXiv preprint arXiv:2405.19722 (2024)

Nguyen, X.-B., Nguyen, H.-Q., Chen, S.Y.-C., Khan, S.U., Churchill, H., Luu, K.: Qclusformer: A quantum transformer-based framework for unsupervised visual clustering. arXiv preprint arXiv:2405.19722 (2024)

2024 arXiv
[57]

https://arxiv.org/abs/2106.07035

Zhao, T., Wang, Z., Masoomi, A., Dy, J.: Deep Bayesian Unsupervised Lifelong Learning (2021). https://arxiv.org/abs/2106.07035

2021 arXiv
[58]

https://arxiv.org/abs/1708.01547

Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong Learning with Dynamically Expandable Networks (2018). https://arxiv.org/abs/1708.01547

2018 arXiv
[59]

https://arxiv.org/abs/1606.04671

Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive Neural Networks (2022). https://arxiv.org/abs/1606.04671

2022 arXiv
[60]

https://arxiv.org/abs/1910.14481

Rao, D., Visin, F., Rusu, A.A., Teh, Y.W., Pascanu, R., Hadsell, R.: Continual Unsupervised Representation Learning (2019). https://arxiv.org/abs/1910.14481

2019 arXiv
[61]

https://arxiv.org/abs/2011.12328

Loo, N., Swaroop, S., Turner, R.E.: Generalized Variational Continual Learning (2020). https://arxiv.org/abs/2011.12328

2020 arXiv
[62]

https://arxiv.org/abs/1904.00310

Li, X., Zhou, Y., Wu, T., Socher, R., Xiong, C.: Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting (2019). https://arxiv.org/abs/1904.00310

2019 arXiv
[63]

https://arxiv.org/abs/2006

Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., Farhadi, A.: Supermasks in Superposition (2020). https://arxiv.org/abs/2006. 14769

2020
[64]

https://arxiv.org/abs/1801

Serr` a, J., Sur ´ ıs, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic for- getting with hard attention to the task (2018). https://arxiv.org/abs/1801. 01423

2018
[65]

https://arxiv.org/abs/1711.05769

Mallya, A., Lazebnik, S.: PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (2018). https://arxiv.org/abs/1711.05769

2018 arXiv
[66]

https://arxiv.org/abs/2112.10017

Ke, Z., Liu, B., Huang, X.: Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021). https://arxiv.org/abs/2112.10017

2021 arXiv
[67]

Nature Machine Intelligence5(10), 1097–1107 (2023)

D´ efossez, A., Caucheteux, C., Rapin, J., Kabeli, O., King, J.-R.: Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence5(10), 1097–1107 (2023)

2023
[68]

In: Image and Graphics: 33 11th International Conference, ICIG 2021, Haikou, China, August 6–8, 2021, Proceedings, Part II 11, pp

Yu, H., Wang, H., Wu, J.: Mixup without hesitation. In: Image and Graphics: 33 11th International Conference, ICIG 2021, Haikou, China, August 6–8, 2021, Proceedings, Part II 11, pp. 143–154 (2021). Springer

2021
[69]

Advances in Neural Information Processing Systems 35, 17612–17625 (2022)

Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems 35, 17612–17625 (2022)

2022
[70]

arXiv preprint arXiv:1905.11946, 6105–6114 (2019)

Tan, M.: Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946, 6105–6114 (2019)

2019 arXiv
[71]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

Caron, M., Touvron, H., Misra, I., J´ egou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660 (2021)

2021
[72]

Science 293(5539), 2425–2430 (2001)

Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P.: Dis- tributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539), 2425–2430 (2001)

2001
[73]

Human brain mapping 8(4), 272–284 (1999)

Fischl, B., Sereno, M.I., Tootell, R.B., Dale, A.M.: High-resolution intersubject averaging and a coordinate system for the cortical surface. Human brain mapping 8(4), 272–284 (1999)

1999
[74]

: A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence

Allen, E.J., St-Yves, G., Wu, Y., Breedlove, J.L., Prince, J.S., Dowdle, L.T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. : A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience 25(1), 116–126 (2022)

2022
[75]

In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–...

2014
[76]

In: European Conference on Computer Vision, pp

Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J.,et al.: Dualprompt: Complementary prompting for rehearsal-free continual learning. In: European Conference on Computer Vision, pp. 631–648 (2022). Springer

2022
[77]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139–149 (2022)

2022
[78]

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision (2021) 34

2021
[79]

arXiv preprint arXiv:2009.02696 (2020)

Martino, G., Barr´ on-Cedeno, A., Wachsmuth, H., Petrov, R., Nakov, P.: Semeval- 2020 task 11: Detection of propaganda techniques in news articles. arXiv preprint arXiv:2009.02696 (2020)

2020 arXiv
[80]

arXiv preprint arXiv:1711.05101 (2017)

Loshchilov, I.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

2017 arXiv
[81]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Douillard, A., Chen, Y., Dapogny, A., Cord, M.: Plop: Learning without for- getting for continual semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4040–4050 (2021)

2021
[82]

IEEE Transactions on Pattern Analysis and Machine Intelligence 40(12), 2935–2947 (2018) https://doi.org/10

Li, Z., Hoiem, D.: Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(12), 2935–2947 (2018) https://doi.org/10. 1109/TPAMI.2017.2773081

2018
[83]

arXiv preprint arXiv:2306.15195 (2023)

Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleash- ing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)

2023 arXiv
[84]

https://arxiv.org/abs/2210.01769

Lin, S., Sprague, T., Singh, A.K.: Mind Reader: Reconstructing complex images from brain activities (2022). https://arxiv.org/abs/2210.01769

2022 arXiv
[85]

https://arxiv.org/abs/2211.06956

Chen, Z., Qing, J., Xiang, T., Yue, W.L., Zhou, J.H.: Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding (2023). https://arxiv.org/abs/2211.06956

2023 arXiv
[86]

https://arxiv

Gu, Z., Jamison, K., Kuceyeski, A., Sabuncu, M.: Decoding natural image stimuli from fMRI data with a surface-based convolutional network (2023). https://arxiv. org/abs/2212.02409

2023 arXiv
[87]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Takagi, Y., Nishimoto, S.: High-resolution image reconstruction with latent dif- fusion models from human brain activity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14453–14463 (2023)

2023
[88]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., Yue, X.: Onellm: One framework to align all modalities with language. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26584–26595 (2024)

2024
[89]

arXiv preprint arXiv:2305.11560 (2023)

Ferrante, M., Ozcelik, F., Boccato, T., VanRullen, R., Toschi, N.: Brain cap- tioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560 (2023)

2023 arXiv
[90]

arXiv preprint arXiv:2308.07428 (2023) 35

Mai, W., Zhang, Z.: Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428 (2023) 35

2023 arXiv
[91]

Journal of machine learning research 9(11) (2008) 36

Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008) 36

2008

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Wang, S., Liu, S., Tan, Z., Wang, X.: Mindbridge: A cross-subject brain decoding framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11333–11342 (2024)

2024

[2] [2]

In: 2022 International Joint Conference on Neural Networks (IJCNN), pp

Ozcelik, F., Choksi, B., Mozafari, M., Reddy, L., VanRullen, R.: Reconstruc- tion of perceived images from fmri patterns and semantic brain exploration using instance-conditioned gans. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2022). IEEE

2022

[3] [3]

PLoS computational biology 15(1), 1006633 (2019) 28

Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS computational biology 15(1), 1006633 (2019) 28

2019

[4] [5]

Advances in Neural Information Processing Systems 36 (2024)

Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., Ver- linde, N., Yundler, E., Weisberg, D., Norman, K., et al.: Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems 36 (2024)

2024

[5] [6]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Quan, R., Wang, W., Tian, Z., Ma, F., Yang, Y.: Psychometry: An omnifit model for image reconstruction from human brain activity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 233–243 (2024)

2024

[6] [7]

arXiv preprint arXiv:2403.18211 (2024)

Huo, J., Wang, Y., Qian, X., Wang, Y., Li, C., Feng, J., Fu, Y.: Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi- level modulation. arXiv preprint arXiv:2403.18211 (2024)

arXiv 2024

[7] [8]

arXiv preprint arXiv:2403.11207 (2024)

Scotti, P.S., Tripathy, M., Villanueva, C.K.T., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K.A., et al.: Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403.11207 (2024)

arXiv 2024

[8] [9]

In: Learning to Learn, pp

Thrun, S.: Lifelong learning algorithms. In: Learning to Learn, pp. 181–209. Springer, ??? (1998)

1998