Pith. sign in

REVIEW 3 major objections 5 minor 64 references

BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activities

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BrainGuard claims that a privacy-preserving global-local framework with hybrid synchronization reconstructs images from multisubject fMRI at state-of-the-art accuracy without ever sharing raw brain data.

desk verdict Federated brain decoding that works, but the privacy claim is unsupported. read the letter →

arxiv 2501.14309 v1 pith:ISYOQ3TK submitted 2025-01-24 cs.CV

Zhibo Tian , Ruijie Quan , Fan Ma , Kun Zhan , Yi Yang This is my paper · ORCID
classification cs.CV
keywords fMRIimagereconstructionbraindecodingprivacy-preservingmachinelearningfederatedmultisubjectmodelinghybridsynchronizationCLIPembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BrainGuard proposes a way to train one image-reconstruction model across multiple people's fMRI scans without ever pooling the raw brain data. Each subject keeps a private local model; a shared global model aggregates only EMA-smoothed model parameters, and a hybrid synchronization strategy decides which layers stay personal, which copy the global model, and which are adaptively blended. The paper reports state-of-the-art reconstruction quality on the Natural Scenes Dataset, including PixCorr 0.313, Inception 96.1%, and CLIP 96.4%, from a single training session across four subjects. The central claim is that privacy and multisubject generalization can be achieved together, and the hybrid layer-wise synchronization is what makes the trade-off work.

What carries the argument

The load-bearing mechanism is the global-local collaborative training loop combined with a hybrid synchronization strategy. The loop runs three steps: each subject trains a private model on local fMRI; individual parameters are EMA-smoothed and aggregated into a global model with subject-size weights; then each local model is updated from the global model in a layer-dependent way. The hybrid strategy has three branches: retention (foundational layers are never updated from global), global alignment (intermediate layers are overwritten by global parameters), and adaptive tuning (advanced layers are fused element-wise via a Dynamic Fusion Learner, which learns blending weights $W_s^m$ in $[0,1]$ by gradient descent on the reconstruction loss). This design is what lets the model balance subject-specific neural signatures against cross-subject commonalities without centralizing data.

What would settle it

An attacker with access to the shared per-subject EMA-smoothed parameters and the global model could try to reconstruct or match a subject's held-out fMRI responses via model inversion or membership inference; if the attacker succeeds beyond chance, the privacy claim is falsified. A concrete test is to train a linear probe on the shared parameters to classify which subject each parameter vector came from, or to attempt to recover binned voxel patterns from the parameters.

Watch

Extended reading notes

Core claim

BRAIN GUARD's central claim is that multisubject fMRI-to-image reconstruction can be made privacy-preserving without sacrificing accuracy: a global-local collaborative framework in which raw fMRI never leaves each subject, combined with a hybrid synchronization strategy, outperforms both subject-specific and centralized multisubject methods. The global model is formed by aggregating EMA-smoothed parameters from individual models with weights proportional to each subject's data size. The hybrid synchronization then transfers global knowledge back selectively: foundational layers are retained locally to preserve subject-specific neural signatures, intermediate layers are globally aligned to capture shared patterns, and advanced layers are adaptively fused through a Dynamic Fusion Learner that learns per-parameter blending weights. On the NSD benchmark, this yields state-of-the-art results across low-level and high-level metrics, e.g., PixCorr .313, SSIM .330, Alex(5) 97.8%, Incept. 96.1%, CLIP 96.4%, with lower distance metrics EffNet-B .624 and SwAV .353, improving on prior multisubject and subject-specific methods.

Load-bearing premise

The privacy guarantee rests on the unstated premise that sharing EMA-smoothed model parameters, rather than raw fMRI, is sufficient to keep sensitive brain data private, a premise the paper never verifies with privacy analysis or attacks.

Editorial extensions

If this is right

  • A single training session across subjects replaces per-subject training runs, so adding a new subject requires only local adaptation, not retraining all previous models.
  • The shared global model can be used as a starting point for new subjects with little data, potentially reducing the hours of fMRI needed for personalized decoding.
  • Because raw fMRI never leaves the local site, multisubject models can be trained across hospitals or labs that cannot share brain scans, enabling larger collaborative datasets.
  • The hybrid synchronization pattern offers a general recipe for other high-dimensional, subject-heterogeneous biosignals, as the paper itself notes for EEG and MEG.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The privacy story is weaker than the phrase 'ensuring privacy preservation' implies: sharing EMA-smoothed model parameters can still leak information about the training data through model-inversion or membership-inference attacks, and the paper provides no differential privacy or secure aggregation analysis. A concrete attack experiment would clarify how much protection 'parameters only' actually
  • The DFL layer-count ablation shows a sweet spot at eight aggregated layers rather than monotone gains, suggesting that lower global-model layers carry generic features while upper layers need subject-specific tuning; testing more network depths or fusion schedules could turn this heuristic into a design rule.
  • Because the global model is a weighted average of EMA-smoothed local parameters, its ability to capture intersubject commonalities is limited to what survives averaging; adding explicit alignment losses or clustering subjects might improve the global model further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes BrainGuard, a federated global-local training framework for multisubject fMRI-to-image reconstruction. Each subject trains a local model on their own fMRI; after local training, per-subject EMA-smoothed parameters are aggregated into a global model, and the global parameters are then re-integrated into local models through a hybrid synchronization strategy: foundational layers are retained, intermediate layers are overwritten by the global model, and advanced layers are fused with the global model using a Dynamic Fusion Learner (DFL) module whose per-layer aggregation weights are learned. Training uses MSE and SoftCLIP losses to align fMRI embeddings with CLIP image/text embeddings, and inference uses a frozen Versatile Diffusion model. The paper claims that this architecture preserves privacy because raw fMRI data is never aggregated, and reports state-of-the-art reconstruction metrics on the NSD dataset (e.g., PixCorr 0.313, Incept. 96.1%, CLIP 96.4% in Table 1).

Significance. If the results hold, the accuracy gains in Table 1 are modest but consistent across most metrics, and the framework is a plausible way to exploit cross-subject commonalities without centralizing raw fMRI. The authors release code, report ablations of the synchronization strategy, and use standard, well-defined losses, which makes the experimental contribution reproducible in principle. The architectural idea of layer-wise hybrid synchronization, borrowing from FedALA, is reasonable and clearly described. However, the paper's central and advertised contribution is privacy preservation, and that claim is currently asserted rather than established. The paper provides no threat model, no differential privacy guarantee, no secure aggregation, and no leakage analysis. The stress-test concern about the privacy claim is therefore valid and load-bearing. The reconstruction accuracy results may be defensible, but the first claimed contribution—privacy—requires either a formal privacy analysis or a substantial reframing of the claims.

major comments (3)
  1. [Abstract and §3.1; Appendix H] The privacy-preservation claim is not established. The paper equates keeping raw fMRI local with 'ensuring privacy preservation' (Abstract, §3.1), but the only mechanism offered is that individual models transmit updated parameters θ'_s and the global model is formed as θ_g = Σ_s k_s θ'_s. These parameters are trained directly on the local fMRI and may encode information about the training data. No threat model is defined, no differential privacy guarantee is provided, no secure aggregation is used, and no analysis or experiment quantifies how much of an individual's fMRI could be recovered from θ'_s or θ_g. Because privacy is the first stated contribution and appears in the paper's title, this is a load-bearing gap. Appendix H lists limitations but omits any privacy caveat, suggesting the property is assumed rather than demonstrated. The authors should either add a formal privacy analysis with a concrete threat model and leakage evaluation, or substantially weaken the privacy claims.
  2. [Table 3 and §C.1] The DFL layer count m is selected on the test set. Table 3 shows that m=8 is chosen as the 'optimal setting' based on test-set performance, and all reported results are single-run with no error bars or variance estimates. This makes the claimed benchmark improvements susceptible to overfitting to the test set and to optimization noise. Additionally, the m=8 row in Table 3 (Alex(2) 95.2%, Alex(5) 98.1%, Incept. 96.3%, CLIP 96.5%, SwAV 0.354) does not exactly match the corresponding BrainGuard results in Table 1 (Alex(2) 94.7%, Alex(5) 97.8%, Incept. 96.1%, CLIP 96.4%, SwAV 0.353) for what appears to be the same configuration. The authors should clarify which result is the final one, report standard deviations or confidence intervals, and use a validation split for selecting m.
  3. [§4.1 and Table 1] The comparison protocol for baselines is underspecified. The paper reports results for eight previous methods, but it does not state whether these numbers are recomputed under the same train/test split and inference protocol (including the retrieval-enhanced inference described in Appendix B) or taken from the original papers. If they are taken from the original papers, the comparison may not be apples-to-apples, especially because the retrieval mechanism can substantially affect high-level metrics. Please clarify how baseline numbers were obtained and, if possible, report results under a unified evaluation protocol.
minor comments (5)
  1. [§4.2, Table 2] The checkmark notation in Table 2 is ambiguous; the text says the first row is BrainGuard without the hybrid synchronization strategy, but the row labels for the checkmarked rows are not explicit. Please label each row clearly (e.g., 'none', 'Found.+Inter.', 'Found.+Inter.+Advan.').
  2. [§4.1, quantitative results text] The text states '31.3%, 33.0%' for PixCorr and SSIM, but these metrics are not percentages; they should be reported as 0.313 and 0.330 to avoid confusion with the percentage-based high-level metrics.
  3. [Appendix B and §4.1] The retrieval-enhanced inference mechanism is described only in the appendix. Since it may significantly affect the reported metrics, it should be described in the main text or at least referenced explicitly when the reconstruction results are presented.
  4. [Introduction] There are typos: 'exiting' should be 'existing' in Section 1, and 'perserving' should be 'preserving' in the contribution list.
  5. [Figure 2] The t-SNE visualization in Figure 2 is informal and not quantified. If the claim is that BrainGuard better aligns subject-specific embeddings, please provide a quantitative alignment metric or a more systematic analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reconstruction pipeline is a standard federated global-local training loop with external benchmarks, while the privacy assertion is a support gap rather than a circular derivation.

full rationale

The paper's derivation chain is self-contained: per-subject models minimize MSE and SoftCLIP losses against CLIP embeddings (Eqs. 4-7), parameters are EMA-smoothed and aggregated into a global model by data-proportion weights, and synchronization uses retention, global alignment, and the DFL update of Eqs. 1-3, which the paper explicitly attributes to FedALA (Zhang et al., 2023). No evaluated metric or benchmark number enters the definitions of the losses, the aggregation rule, or the objective in Eq. 8, so no reported result is forced by construction. The authors cite their own prior Psychometry paper (Quan et al., 2024) as a comparison baseline in Table 1 and related work, but the architecture and experiments do not depend on that citation; the DFL mechanism is credited to an external source and the comparisons are measured against independent baselines. The privacy claim in the Abstract and Section 3.1 equates keeping raw fMRI local with privacy preservation and lacks a threat model, differential privacy guarantee, secure aggregation, or leakage analysis. This is a real support gap and should be weighed as a correctness or completeness concern, but it is not a circular step: no equation or prediction is defined in terms of the privacy property, and the reconstruction results stand or fall independently of that unsupported assertion. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method inherits standard ML assumptions, such as CLIP as target space and diffusion priors, and introduces one strong unsupported premise: that local data retention plus parameter sharing guarantees privacy. The main data-dependent free parameter is the DFL layer count m, selected on the test set. No new physical or conceptual entities are invented; the global model and DFL module are assembled from prior work.

free parameters (4)
  • DFL layer count m = 8 (selected on NSD test set, Table 3)
    Appendix C.1 reports an ablation over m from 1 to 12 and picks m=8 by test-set metrics; this is a free parameter tuned on the evaluation set, affecting all headline results.
  • EMA factor alpha = 0.999
    Set in §3.1 to smooth local updates; no sensitivity analysis is provided.
  • SoftCLIP temperature tau = not specified in paper
    Equation (5) uses tau but no value is given; the result depends on this contrastive scaling hyperparameter.
  • Layer-group boundaries (foundational/intermediate/advanced) = not specified
    The paper defines three synchronization parts but does not state which layers fall in each group beyond 'm advanced layers'; this structural choice is a free design parameter.
assumptions (3)
  • domain assumption Sharing only model parameters while keeping raw fMRI local is sufficient to preserve privacy.
    Stated in Abstract and §3.1 as 'thereby ensuring privacy preservation', but no differential privacy, secure aggregation, or information-leakage analysis is provided; parameters can encode training data.
  • ad hoc to paper The global model, formed by weighted averaging of individual model parameters, provides a useful shared representation for all subjects.
    Assumed in §3.1 step 2; FedAvg-style averaging is known to work for some distributions but is not guaranteed for highly heterogeneous fMRI data, and no theoretical or empirical analysis of the global model's quality is presented beyond Fig. 5(a).
  • domain assumption CLIP embeddings of images and captions are a sufficient target space for fMRI-to-image reconstruction.
    Inherited from MindEye, Brain-Diffuser, and related methods; the paper uses CLIP as the shared latent space without questioning its completeness for visual reconstruction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activities." pith.science (2026). https://pith.science/paper/ISYOQ3TK

@misc{pith2026250114309,
  author       = {Pith},
  title        = {Pith review of: BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ISYOQ3TK}},
  note         = {Machine review of arXiv:2501.14309}
}
read the original abstract

Reconstructing perceived images from human brain activity forms a crucial link between human and machine learning through Brain-Computer Interfaces. Early methods primarily focused on training separate models for each individual to account for individual variability in brain activity, overlooking valuable cross-subject commonalities. Recent advancements have explored multisubject methods, but these approaches face significant challenges, particularly in data privacy and effectively managing individual variability. To overcome these challenges, we introduce BrainGuard, a privacy-preserving collaborative training framework designed to enhance image reconstruction from multisubject fMRI data while safeguarding individual privacy. BrainGuard employs a collaborative global-local architecture where individual models are trained on each subject's local data and operate in conjunction with a shared global model that captures and leverages cross-subject patterns. This architecture eliminates the need to aggregate fMRI data across subjects, thereby ensuring privacy preservation. To tackle the complexity of fMRI data, BrainGuard integrates a hybrid synchronization strategy, enabling individual models to dynamically incorporate parameters from the global model. By establishing a secure and collaborative training environment, BrainGuard not only protects sensitive brain data but also improves the image reconstructions accuracy. Extensive experiments demonstrate that BrainGuard sets a new benchmark in both high-level and low-level metrics, advancing the state-of-the-art in brain decoding through its innovative design.

Figures

Figures reproduced from arXiv: 2501.14309 by the authors.

Figure 1
Figure 1. (a) Early subject-specific methods require separate [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) t-SNE visualization of the embeddings output [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An overview of the BRAINGUARD training and inference framework (§3.1). across subjects. Before this aggregation, individual models update their parameters using the Exponential Moving Av￾erage (EMA): θ ′ s = αθ′ s + (1 − α)θs, where α is the EMA factor, typically set to 0.999. This step aims to smooth param￾eter changes and enhance individual model stability, ensuring that the subsequent aggregation reflects both re… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons on the NSD test dataset. BRAINGUARD performs a single training session on multisubject fMRI data, demonstrates superior reconstruction accuracy compared to four recent state-of-the-art methods (Quan et al. 2024; Wang et al. 2024; Scotti et al. 2…
Figure 6
Figure 6. Figure 6: For quantitative evaluation, we present eight differ [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 5
Figure 5. Figure 5: (a) We evaluate our trained global model and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Examples of fMRI reconstructions generated by [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Additional resconstruction results on NSD [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 55 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    J.; St-Yves, G.; Wu, Y.; Breedlove, J

    Allen, E. J.; St-Yves, G.; Wu, Y.; Breedlove, J. L.; Prince, J. S.; Dowdle, L. T.; Nau, M.; Caron, B.; Pestilli, F.; Charest, I.; et al. 2022. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1): 116--126

    work page 2022

  4. [4]

    Beliy, R.; Gaziv, G.; Hoogi, A.; Strappini, F.; Golan, T.; and Irani, M. 2019. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. NeurIPS, 32

    work page 2019

  5. [5]

    Briggs, C.; Fan, Z.; and Andras, P. 2020. Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In IJCNN, 1--9

    work page 2020

  6. [6]

    Bullmore, E.; and Sporns, O. 2009. Complex brain networks: graph theoretical analysis of structural and functional systems. Nature Reviews Neuroscience, 10(3): 186--198

    work page 2009

  7. [7]

    Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; and Joulin, A. 2020. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, volume 33, 9912--9924

    work page 2020

  8. [8]

    A.; Marcus, A.; Gupta, A.; Tarr, M

    Chang, N.; Pyles, J. A.; Marcus, A.; Gupta, A.; Tarr, M. J.; and Aminoff, E. M. 2019. BOLD5000, a public fMRI dataset while viewing 5000 visual images. Scientific data, 6(1): 49

    work page 2019

Show all 64 references
  1. [9]

    Chen, Y.; Qin, X.; Wang, J.; Yu, C.; and Gao, W. 2020. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 35(4): 83--93

    2020

  2. [10]

    L.; and Zhou, J

    Chen, Z.; Qing, J.; Xiang, T.; Yue, W. L.; and Zhou, J. H. 2023. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In CVPR

    2023

  3. [11]

    Courbariaux, M.; Hubara, I.; Soudry, D.; El-Yaniv, R.; and Bengio, Y. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830

    2016 arXiv

  4. [12]

    D.; and Savoy, R

    Cox, D. D.; and Savoy, R. L. 2003. Functional magnetic resonance imaging ( fMRI ) ``brain reading'': Detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage, 19(2): 261--270

    2003

  5. [13]

    Fallah, A.; Mokhtari, A.; and Ozdaglar, A. 2020. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In NeurIPS, volume 33, 3557--3568

    2020

  6. [14]

    Ghosh, A.; Chung, J.; Yin, D.; and Ramchandran, K. 2020. An efficient framework for clustered federated learning. In NeurIPS, volume 33, 19586--19597

    2020

  7. [15]

    Gong, Z.; Zhang, Q.; Bao, G.; Zhu, L.; Liu, K.; Hu, L.; and Miao, D. 2024. MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction. arXiv preprint arXiv:2404.12630

    2024 arXiv

  8. [16]

    Gu, Z.; Jamison, K.; Kuceyeski, A.; and Sabuncu, M. 2023. Decoding natural image stimuli from fMRI data with a surface-based convolutional network. In MIDL

    2023

  9. [17]

    He, C.; Annavaram, M.; and Avestimehr, S. 2020. Group knowledge transfer: Federated learning of large cnns at the edge. In NeurIPS, volume 33, 14068--14080

    2020

  10. [18]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. In NeurIPS, volume 33, 6840--6851

    2020

  11. [19]

    Horikawa, T.; and Kamitani, Y. 2017. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1): 15037

    2017

  12. [20]

    Kamitani, Y.; and Tong, F. 2005. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5): 679--685

    2005

  13. [21]

    Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020. Analyzing and improving the image quality of stylegan. In CVPR, 8110--8119

    2020

  14. [22]

    N.; Naselaris, T.; Prenger, R

    Kay, K. N.; Naselaris, T.; Prenger, R. J.; and Gallant, J. L. 2008. Identifying natural images from human brain activity. Nature, 452(7185): 352--355

    2008

  15. [23]

    Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In NeurIPS, volume 25

    2012

  16. [24]

    K.; Belliveau, J

    Kwong, K. K.; Belliveau, J. W.; Chesler, D. A.; Goldberg, I. E.; Weisskoff, R. M.; Poncelet, B. P.; Kennedy, D. N.; Hoppel, B. E.; Cohen, M. S.; and Turner, R. 1992. Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. PNAS, 89(12): 5675--5679

    1992

  17. [25]

    LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. Nature, 521(7553): 436--444

    2015

  18. [26]

    Li, D.; and Wang, J. 2019. FedMD : Heterogenous federated learning via model distillation. In NeurIIPS Workshop

    2019

  19. [27]

    Li, Q.; He, B.; and Song, D. 2021. Model-contrastive federated learning. In CVPR, 10713--10722

    2021

  20. [28]

    K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V

    Li, T.; Sahu, A. K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2: 429--450

    2020

  21. [29]

    Li, Y.; Quan, R.; Zhu, L.; and Yang, Y. 2023. Efficient multimodal fusion via interactive prompting. In CVPR, 2604--2613

    2023

  22. [30]

    Lin, S.; Sprague, T.; and Singh, A. K. 2022. Mind reader: Reconstructing complex images from brain activities. NeurIPS, 35: 29624--29636

    2022

  23. [31]

    U.; and Jaggi, M

    Lin, T.; Kong, L.; Stich, S. U.; and Jaggi, M. 2020. Ensemble distillation for robust model fusion in federated learning. In NeurIPS, volume 33, 2351--2363

    2020

  24. [32]

    Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In ECCV, 740--755

    2014

  25. [33]

    Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, volume 32

    2019

  26. [34]

    Mai, W.; Zhang, J.; Fang, P.; and Zhang, Z. 2023. Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy. arXiv preprint arXiv:2401.00430

    2023 arXiv

  27. [35]

    McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 1273--1282

    2017

  28. [36]

    C.; Sadato, N.; and Kamitani, Y

    Miyawaki, Y.; Uchida, H.; Yamashita, O.; Sato, M.-a.; Morito, Y.; Tanabe, H. C.; Sadato, N.; and Kamitani, Y. 2008. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, 60(5): 915--929

    2008

  29. [37]

    T.; Naselaris, T.; Benjamini, Y.; Yu, B.; and Gallant, J

    Nishimoto, S.; Vu, A. T.; Naselaris, T.; Benjamini, Y.; Yu, B.; and Gallant, J. L. 2011. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19): 1641--1646

    2011

  30. [38]

    Ozcelik, F.; and VanRullen, R. 2023. Natural scene reconstruction from fMRI signals using generative latent diffusion. Scientific Reports, 13(1): 15666

    2023

  31. [39]

    Parthasarathy, N.; Batty, E.; Falcon, W.; Rutten, T.; Rajpal, M.; Chichilnisky, E.; and Paninski, L. 2017. Neural networks for efficient bayesian decoding of natural images from retinal neurons. In NeurIPS, volume 30

    2017

  32. [40]

    Quan, R.; Wang, W.; Tian, Z.; Ma, F.; and Yang, Y. 2024. Psychometry: An omnifit model for image reconstruction from human brain activity. In CVPR, 233--243

    2024

  33. [41]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748--8763

    2021

  34. [42]

    Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021. Zero-shot text-to-image generation. In ICML, 8821--8831

    2021

  35. [43]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In CVPR, 10684--10695

    2022

  36. [44]

    Sattler, F.; M \"u ller, K.-R.; and Samek, W. 2020. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8): 3710--3722

    2020

  37. [45]

    S.; Banerjee, A.; Goode, J.; Shabalin, S.; Nguyen, A.; Cohen, E.; Dempster, A

    Scotti, P. S.; Banerjee, A.; Goode, J.; Shabalin, S.; Nguyen, A.; Cohen, E.; Dempster, A. J.; Verlinde, N.; Yundler, E.; Weisberg, D.; et al. 2023. Reconstructing the Mind's Eye: fMRI -to-Image with Contrastive Learning and Diffusion Priors. In NeurIPS, volume 36

    2023

  38. [46]

    S.; Tripathy, M.; Villanueva, C

    Scotti, P. S.; Tripathy, M.; Villanueva, C. K. T.; Kneeland, R.; Chen, T.; Narang, A.; Santhirasegaran, C.; Xu, J.; Naselaris, T.; Norman, K. A.; et al. 2024. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. arXiv preprint arXiv:2403.11207

    2024 arXiv

  39. [47]

    Shen, G.; Horikawa, T.; Majima, K.; and Kamitani, Y. 2019. Deep image reconstruction from human brain activity. PLoS Computational Biology, 15(1): e1006633

    2019

  40. [48]

    Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2256--2265

    2015

  41. [49]

    Song, Y.; and Ermon, S. 2019. Generative modeling by estimating gradients of the data distribution. In NeurIPS, volume 32

    2019

  42. [50]

    P.; Kumar, A.; Ermon, S.; and Poole, B

    Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2020. Score-based generative modeling through stochastic differential equations. In ICLR

    2020

  43. [51]

    Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016. Rethinking the inception architecture for computer vision. In CVPR, 2818--2826

    2016

  44. [52]

    Takagi, Y.; and Nishimoto, S. 2023. High-resolution image reconstruction with latent diffusion models from human brain activity. In CVPR, 14453--14463

    2023

  45. [53]

    Tan, M.; and Le, Q. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 6105--6114

    2019

  46. [54]

    C.; Smith, S

    Van Essen, D. C.; Smith, S. M.; Barch, D. M.; Behrens, T. E.; Yacoub, E.; Ugurbil, K.; Consortium, W.-M. H.; et al. 2013. The WU-Minn human connectome project: an overview. Neuroimage, 80: 62--79

    2013

  47. [55]

    Wang, S.; Liu, S.; Tan, Z.; and Wang, X. 2024. Mindbridge: A cross-subject brain decoding framework. In CVPR, 11333--11342

    2024

  48. [56]

    C.; Sheikh, H

    Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on Image Processing, 13(4): 600--612

    2004

  49. [57]

    Xu, X.; Wang, Z.; Zhang, G.; Wang, K.; and Shi, H. 2023. Versatile diffusion: Text, images and variations all in one diffusion model. In CVPR

    2023

  50. [58]

    Yang, H.; He, H.; Zhang, W.; and Cao, X. 2020. FedSteg: A federated transfer learning framework for secure image steganalysis. IEEE Transactions on Network Science and Engineering, 8(2): 1084--1094

    2020

  51. [59]

    Yang, Z.; Chen, G.; Li, X.; Wang, W.; and Yang, Y. 2024. Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent). In ICML

    2024

  52. [60]

    Yao, X.; and Sun, L. 2020. Continual local training for better initialization of federated models. In ICIP, 1736--1740

    2020

  53. [61]

    Yosinski, J.; Clune, J.; Bengio, Y.; and Lipson, H. 2014. How transferable are features in deep neural networks? In NeurIPS, volume 27

    2014

  54. [62]

    Zhang, J.; Hua, Y.; Wang, H.; Song, T.; Xue, Z.; Ma, R.; and Guan, H. 2023. FedALA : Adaptive local aggregation for personalized federated learning. In AAAI, volume 37, 11237--11244

    2023

  55. [63]

    Zhao, S.; Quan, R.; Zhu, L.; and Yang, Y. 2024. CLIP4STR: A simple baseline for scene text recognition with pre-trained vision-language model. IEEE transactions on Image Processing

    2024

  56. [64]

    Zhu, Z.; Hong, J.; and Zhou, J. 2021. Data-free knowledge distillation for heterogeneous federated learning. In ICML, 12878--12889

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.