Pith. sign in

REVIEW 4 major objections 4 minor 77 references

fMRI2Face: A Full-HD fMRI-Video Dataset and Geometry-Guided Neural Decoding Framework for Dynamic Human Face Reconstruction

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces fMRI-Face, the first fMRI dataset paired with controllable full-HD (1920×1080) digital human facial videos, and fMRI2Face, a geometry-guided framework that reconstructs dynamic facial videos from brain activity while p

desk verdict Valuable dataset and a plausible decoder, but the held-out set reuses the same face shapes and motions as training, so the headline decoding claim is not yet supported. read the letter →

arxiv 2607.22302 v1 pith:5NQ4TUSV submitted 2026-07-24 cs.CV

Jingyang Huo , Xiangru Huang , Chentao Shen , Yikai Wang , Yun Wang , Jianxiong Gao , Shihao Jin , Yanwei Fu
show 1 more author
Jianfeng Feng
This is my paper · ORCID
classification cs.CV
keywords neuraldecodingfMRIdatasetfacereconstructiondigitalhumanvideogeneration3Dmorphablemodeldiffusionperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that dynamic human faces can be reconstructed from fMRI signals at full-HD resolution when the decoder is given explicit 3D face geometry as a guide. To make this possible it introduces fMRI-Face, a dataset of 62,856 paired fMRI-video samples in which three participants viewed background-free, full-HD digital human faces with controlled identity, expression, and head pose. Its fMRI2Face framework splits decoding into two neural controls: appearance context tokens carrying global identity-related attributes, and morphable 3D facial parameters rendered into geometry-consistent motion guidance, both injected into a pretrained video diffusion model. On every reported metric the framework outperforms three representative decoding baselines, sharply improving identity similarity and temporal coherence. If the result holds, the field gains a controlled benchmark for dynamic face perception and a usable path from brain signals to controllable digital-human video.

What carries the argument

Two complementary fMRI-derived controls carry the argument. Brain-derived Appearance Context is a set of 64 learnable query tokens that a transformer decoder extracts from an fMRI window and feeds into the diffusion transformer's context-conditioning interface, replacing text prompts with brain-derived appearance cues. Morphable 3D Facial Control predicts DECA morphable-face parameters—a standard parametric face model—from the same fMRI window, renders them into geometry-consistent facial control frames, and injects them through a parallel control branch with zero-initialized residual bridges. An auxiliary latent predictor completes visual context outside the rendered face region. The diffus

What would settle it

Compare fMRI2Face's reconstructions against a nearest-neighbor baseline that retrieves the most similar training-video latent for each test fMRI window. If the retrieval baseline matches or beats the reported PSNR, ID-CSIM, and FVD on test clips whose identity or motion type also appears in training, the decoding claim collapses; retraining with identities and motions held out by construction would settle it.

Watch

Extended reading notes

Core claim

The central claim is that fMRI responses from visual cortex carry enough information to reconstruct a perceived digital human's identity and motion, provided the decoder separates appearance from geometry. fMRI2Face predicts fMRI-conditioned context tokens for global appearance and, in parallel, a sequence of DECA morphable-face parameters (shape, expression, pose, albedo, detail, illumination, image transform) that are rendered into a 2D facial guidance video. A frozen video-diffusion transformer is conditioned on both streams through zero-initialized residual bridges and an auxiliary latent completion, producing 480×832 video clips at 15 fps. The paper reports consistent gains over baselin

Load-bearing premise

The held-out test videos are independent of the training videos, so reported scores measure decoding of unseen faces rather than retrieval from a small stimulus manifold built from 31 shared motion types and a shared base-mesh library.

Editorial extensions

If this is right

  • fMRI-Face provides a controlled benchmark that lets future work isolate how identity, expression, and pose are encoded in visual cortex.
  • If the reported gains hold, explicit parametric 3D face modeling should become a standard component of fMRI-to-face decoders.
  • The framework shows that a pretrained video diffusion model can be steered entirely by brain-derived conditions, extending fMRI-conditioned video generation beyond faces.
  • The ROI analyses suggest early visual areas and motion-selective areas differentially drive appearance versus geometry, giving testable predictions for neuroscience.
  • The pipeline points toward controllable digital-human animation from brain activity, with potential as a communication channel for people who cannot speak or move.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state that the 180 held-out videos use identities and motion types disjoint from training; a test split that excludes them by construction would establish whether the decoder generalizes to unseen faces or interpolates within the 31 shared motion configurations.
  • Because stimuli are background-free digital humans under fixed lighting, the dataset measures decoding inside a synthetic stimulus universe; transferring the same architecture to natural face videos or photographs would test its generality.
  • The residual design that anchors identity-related parameters to the first frame suggests a clean cross-subject test: ablating the appearance stream should still leave shape and albedo-derived identity cues in the geometry stream, revealing which stream actually carries identity.
  • The authors list rapid micro-motions as a limitation; combining fMRI with a faster neural recording modality could feed the geometry-control stream with temporal priors and likely recover blinks and subtle expression changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces fMRI-Face, a large fMRI dataset of 62,856 paired samples obtained while three participants watched 2,174 controlled 8-second full-HD digital-human facial videos, and fMRI2Face, a decoding framework that predicts appearance-context tokens and DECA-based 3D facial parameters from fMRI and uses them to condition a pretrained video diffusion transformer. Quantitative results on a held-out set of 180 videos show consistent improvements over MindVideo, NeuroPictor, and MindEye2 across reconstruction, identity, geometry, and motion metrics. The paper also presents ablations, ROI analyses, failure cases, and supplementary details on the stimulus-generation pipeline.

Significance. If the reported results hold, fMRI-Face would be a large, controlled benchmark for dynamic face decoding, and fMRI2Face would demonstrate a useful integration of 3D morphable face control with video diffusion for fMRI-conditioned reconstruction. The paper's strengths are the scale and controllability of the stimulus set, the explicit two-stream design (appearance context plus 3D control), the subject-wise results, and the careful loss ablations for the 3D stream. However, the central generalization claim is not yet adequately supported because the test split may not be independent of the training manifold, and the comparisons lack statistical uncertainty.

major comments (4)
  1. [Sec. 3.2.4 / Supp. 1.1] The held-out set does not establish independence of test identities/motions. Supp. 1.1 reports only 31 motion configurations (16 female, 15 male) for all 2,174 videos, 136 fixed face shapes for the 1,694 artistic digital humans, and 480 parametric videos generated by interpolating a shared base-mesh library with fixed hairstyles and skin textures. A random holdout of 180 videos therefore almost certainly includes test videos whose face shape and motion type already appear in training. Under these conditions, the appearance-context tokens and the DECA 3D control can retrieve or interpolate familiar shape/motion combinations; the gains in Tab. 2 (e.g., PSNR 18.32 vs 15.71, FVD 82.7 vs 361.3) need not reflect fMRI decoding of unseen identities or dynamics. The authors should specify whether test videos are disjoint in shape and motion by construction, or re-split/re-evaluate with disjoint s
  2. [Sec. 5.3.1 / Tabs. 2 and 6] The headline comparisons are reported as single means without error bars or significance tests. Given only 180 test videos and three subjects, metrics such as FVD and ID-CSIM have substantial sampling noise; Tab. 6 shows subject-level variation but no variance within subject. To support "consistently improves," the authors should report confidence intervals (e.g., bootstrap over test clips) and/or paired significance tests against each baseline, and provide subject-wise baseline numbers rather than only our method.
  3. [Sec. 5.4] The ablations are run only on Subject 1, while the main quantitative claims are means over three subjects (Tab. 6). The contribution of each component (context-token count, 3D control, auxiliary latent, mask) may be subject-dependent. Please provide per-subject ablations, or at least justify why Subject 1 is representative; otherwise the design conclusions are not supported on the full dataset.
  4. [Abstract / Sec. 3] The paper's first contribution is the fMRI-Face dataset, but no data availability statement, URL, or code release is included. Without a concrete release plan, the benchmark cannot be used by the community and the central numbers cannot be independently verified. A dataset paper should state the intended availability and license.
minor comments (4)
  1. [Sec. 5.1] The paper describes a full-HD dataset but reconstructs at 480×832. Clarify whether ground-truth videos are downsampled before metric computation and how landmark metrics are scaled across resolutions.
  2. [Sec. 3.2.4] The relationship between 2,174 unique videos, 154 repeated clips, and the reported 2,012 training / 316 test pairs is ambiguous. Specify how repeats were allocated between train and test.
  3. [Figs. 5 and 12] The qualitative figures would benefit from showing the rendered 3D control stream for the same examples, and from including baseline failure cases for comparison.
  4. [Sec. 4.4.3] Parameter-level supervision uses DECA parameters extracted from the GT videos; since DECA is trained on real faces, the paper should report DECA accuracy on digital-human stimuli or otherwise justify the reliability of this supervision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: fMRI2Face's components are supervised on ground-truth labels and evaluated on held-out videos, with no claim reducing to its own inputs by construction.

full rationale

The paper's derivation chain is a standard supervised pipeline: fMRI inputs are mapped to appearance-context tokens and DECA-based geometric controls, which condition a pretrained video diffusion model; reconstructed videos are then compared with ground-truth videos using PSNR, ID-CSIM, LMD, FVD, etc. The Morphable 3D Facial Control stream is trained with parameter-level and landmark-reprojection losses computed from ground-truth DECA parameters and landmarks (Sec. 4.4.3, Eqs. 11-18). This means that the control stream is optimized to reproduce the very landmarks used in the LMD metric, so some of the reported geometry improvement is aligned with the training objective. However, this is not circular by construction: the final videos are generated by the diffusion model rather than being the rendered controls themselves, the metrics are computed on held-out test videos, and the method is compared against baselines under the same train-test split (Tab. 2 caption). The dataset's shared motion configurations and base-mesh library (Supp. 1.1) raise a legitimate concern about whether the holdout measures truly novel identities/motions, but that is a test-set independence / generalization issue, not a case of a fitted parameter being renamed as a prediction or of an equation reducing to its input. Self-citations, e.g., NeuroPictor [30], are used only as an evaluation baseline and not as load-bearing justification. There is no imported uniqueness theorem, no ansatz smuggled via self-citation, and no renaming of a known result as a new derivation. Therefore no specific circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claims rest on domain assumptions about fMRI-to-face information, train/test independence, DECA sufficiency, and digital-human-to-real-face transfer; no new physical entities are introduced. The main free parameters are temporal alignment, token count, guidance scale, and loss weights.

free parameters (5)
  • fMRI-to-clip temporal alignment (5–9 TR delay window) = TR 5–9 after clip onset
    Chosen from known hemodynamic lag but not validated per subject; directly defines the input F∈R^{5×8921} in Sec. 4.1.
  • Number of appearance-context tokens = 64
    Ablation (Tab. 3) shows 1 token fails and 16 < 64; the number materially affects reconstruction quality. No principled criterion is given.
  • Classifier-free guidance scale = 5.0
    Fixed at inference (Sec. 4.5.4); no ablation or sensitivity analysis reported.
  • Loss-balancing coefficients for 3D control = λ_param=1, λ_lmk=1, λ_lip=0.5, λ_eye=1, λ_Δparam=1, λ_Δlmk=1
    Hand-set in Supp. 3.1; no systematic tuning or sensitivity is shown.
  • LoRA rank and clip length = rank 32; 24 frames at 15 fps
    Architecture choices that influence capacity and temporal coverage; reported without ablations.
assumptions (6)
  • domain assumption BOLD responses in the selected 31 visual ROIs (8,921 vertices) carry sufficient information about facial identity, expression, and pose to drive video reconstruction.
    Core premise of the decoding pipeline; if false, observed performance must come from diffusion priors. Invoked in Sec. 3.2.3 and Sec. 4.1.
  • domain assumption The 5–9 TR delayed fMRI window corresponds to the viewed 2-TR clip and is not confounded by adjacent clips.
    A fixed hemodynamic-delay assumption; no HRF deconvolution is performed. Stated in Sec. 4.1.
  • domain assumption Held-out 180 videos are independent from training videos in identity and motion.
    If test videos share identities or the 31 motion configurations with training, metrics may reflect interpolation on a known stimulus manifold, not unseen-face decoding. See Sec. 3.2.4 and Supp. 1.1.
  • domain assumption Digital human stimuli engage the same face-processing mechanisms as real faces.
    The paper argues this via landmark-distribution overlap with MEAD and FFC reliability, but does not directly validate neural equivalence. Supp. 1.2, Sec. 3.3.
  • domain assumption DECA parameters and renderings are a sufficient geometry representation for facial identity and motion.
    The pipeline and landmark metrics depend on DECA's pretrained face model; any systematic DECA errors propagate into the rendered control and evaluation. Sec. 4.4.
  • domain assumption The frozen Wan2.1-T2V-14B video diffusion prior is a suitable generator and is not a hidden source of target information.
    The method relies on a large pretrained generative prior; no analysis separates prior-driven generation from fMRI-driven content. Sec. 5.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of fMRI2Face: A Full-HD fMRI-Video Dataset and Geometry-Guided Neural Decoding Framework for Dynamic Human Face Reconstruction." pith.science (2026). https://pith.science/paper/5NQ4TUSV

@misc{pith2026260722302,
  author       = {Pith},
  title        = {Pith review of: fMRI2Face: A Full-HD fMRI-Video Dataset and Geometry-Guided Neural Decoding Framework for Dynamic Human Face Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NQ4TUSV}},
  note         = {Machine review of arXiv:2607.22302}
}
abstract

Reconstructing dynamic human faces from brain activity provides a powerful way to study how the mind perceives identity, expression, and facial motion. However, progress in fMRI-based face decoding has been limited by scarce controlled, high-resolution neural datasets and by methods that struggle to recover both identity-specific appearance and time-varying facial dynamics. We present fMRI-Face, the first fMRI dataset paired with controllable full-HD digital human facial videos rendered at 1920$\times$1080 resolution. During scanning, participants watched photorealistic, background-free facial videos with controlled identity, expression, and head pose, while fMRI activity was recorded. The resulting dataset contains 62,856 paired fMRI-video samples, providing a structured resource for studying dynamic face perception and reconstruction. Building on this dataset, we propose fMRI2Face, a geometry-guided neural video decoding framework for reconstructing facial videos from fMRI signals. fMRI2Face derives two complementary neural controls from brain activity: Brain-derived Appearance Context, which captures global identity-related visual attributes, and Morphable 3D Facial Control, which provides explicit geometry-aware guidance for pose, expression, and non-rigid facial dynamics. These controls are integrated through Neural-Controlled Video Diffusion with auxiliary latent completion, enabling high-fidelity facial video reconstruction directly from brain activity. Experiments show that fMRI2Face consistently improves reconstruction fidelity, identity preservation, facial geometry, and motion consistency over representative neural decoding baselines. Together, fMRI-Face and fMRI2Face establish a controlled platform for studying dynamic face perception and provide a new benchmark for fMRI-based digital human reconstruction.

Figures

Figures reproduced from arXiv: 2607.22302 by the authors.

Figure 1
Figure 1. Illustration of the fMRI-Face dataset and the fMRI2Face decoding framework. Top: Participants view digital human facial stimuli during fMRI scanning, producing brain responses. Bottom: Given recorded fMRI signals, fMRI2Face derives two complementary neural controls: Brain-derived Ap￾pearance Context for global visual appearance and Morphable 3D Facial Control for geometry-aware structure and motion. The two controls… view at source ↗
Figure 2
Figure 2. Overview of the fMRI-Face dataset workflow. Starting from 3D digital-human creation using modular digital assets, a facial rigging system is configured for each character. Predefined animation controllers drive facial expressions over time, and rendering software produces photorealistic videos for each digital human. Participants view these videos during fMRI scanning. 3.1. Digital Human Video Generation Pipeline Th… view at source ↗
Figure 3
Figure 3. (a) Neural reliability heatmap across the cortex based on repeated video clips; cyan contours denote selected VC ROIs and black contours denote HCP region boundaries. (b) Number of top 50% correlated vertices in each visual ROI. (c) Proportion of top 50% correlated vertices in each visual ROI. 2 TRs (1.6 s). This preprocessing yields 18,108 fMRI–video clips for training and 2,844 for testing per subject. Aggregat￾in… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Overview of the fMRI2Face framework. Left: Given an fMRI input, Morphable 3D Facial Control predicts DECA parameters through the Geometric Predictor geo, decodes them into a 3D facial mesh, and renders a geometry-consistent facial guidance sequence ̂𝐈𝑟 . In parallel, …
Figure 5
Figure 5. Figure 5: Qualitative comparison of facial video reconstruction from fMRI signals. For each example, four frames are shown from left to right in temporal order. Compared with baseline neural decoding methods, fMRI2Face better preserves identity-related appearance, pose, and expr…
Figure 6
Figure 6. Figure 6: Gradient-based ROI attribution for the Brain-derived Appearance Context and Morphable 3D Facial Control streams. (a) The first three heatmaps show gradient attribution standardized within each subject, and the fourth shows the mean of these scores across subjects. (b) …
Figure 7
Figure 7. Figure 7: Single-step diffusion sensitivity to ROI masking. For each ROI, the grouped bars show the mean across subjects and the scores of subjects 1–3. Larger values indicate greater changes in the velocity prediction after masking ROI signals. attribution is computed as 𝐴 𝑏 𝑖,…
Figure 8
Figure 8. Figure 8: Visualization of canonical portraits [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Sample visualization of the 47 fine-grained clothing items used for Parametric Digital Humans [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Landmark-coordinate distribution comparison be￾tween fMRI-Face and MEAD. We extract 68 2D facial landmarks from both datasets and visualize their normalized coordinate distributions using violin plots. The upper and lower panels show the 𝑥- and 𝑦-coordinate distributi…
Figure 11
Figure 11. Figure 11: Subject-wise visualization of reconstructed digital human facial videos. Page 20 of 22 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Visualizations for failure cases. parameter-level supervision provides the basic 3D facial representation, landmark reprojection improves geometric alignment, fine-grained eye/lip losses refine local facial de￾tails, and temporal motion losses enhance dynamic facial c…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 7 linked inside Pith

  1. [1]

    A massive7tfmridatasettobridgecognitiveneuroscienceandartificial intelligence

    Allen,E.J.,St-Yves,G.,Wu,Y.,Breedlove,J.L.,Prince,J.S.,Dowdle, L.T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al., 2022. A massive7tfmridatasettobridgecognitiveneuroscienceandartificial intelligence. Nature neuroscience 25, 116–126

    2022

  2. [2]

    Amorphablemodelforthesynthesisof3d faces, in: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp

    Blanz,V.,Vetter,T.,2023. Amorphablemodelforthesynthesisof3d faces, in: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 157–164

    2023

  3. [3]

    Bold5000, a public fmri dataset while viewing 5000 visual images

    Chang, N., Pyles, J.A., Marcus, A., Gupta, A., Tarr, M.J., Aminoff, E.M., 2019. Bold5000, a public fmri dataset while viewing 5000 visual images. Scientific data 6, 49

    2019

  4. [4]

    Chang, P.C., Tien, Y.Y., Chen, C.L., Chen, L.F., Chen, Y.S., Chan, H.L., 2022. Facial image reconstruction from functional magnetic resonance imaging via gan inversion with improved attribute consis- tency, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8

    2022

  5. [5]

    Videocrafter2: Overcoming data limitations for high-quality videodiffusionmodels,in:ProceedingsoftheIEEE/CVFconference on computer vision and pattern recognition, pp

    Chen,H.,Zhang,Y.,Cun,X.,Xia,M.,Wang,X.,Weng,C.,Shan,Y., 2024a. Videocrafter2: Overcoming data limitations for high-quality videodiffusionmodels,in:ProceedingsoftheIEEE/CVFconference on computer vision and pattern recognition, pp. 7310–7320

  6. [6]

    An fmri dataset in response to large-scale short natural dynamic facial expression videos

    Chen, P., Zhang, C., Li, B., Tong, L., Wang, L., Ma, S., Cao, L., Yu, Z., Yan, B., 2024b. An fmri dataset in response to large-scale short natural dynamic facial expression videos. Scientific Data 11, 1247

  7. [7]

    Chen, Z., Qing, J., Xiang, T., Yue, W.L., Zhou, J.H., 2023a. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.22710– 22720

  8. [8]

    Cinematic mindscapes: High- quality video reconstruction from brain activity

    Chen, Z., Qing, J., Zhou, J.H., 2023b. Cinematic mindscapes: High- quality video reconstruction from brain activity. Advances in Neural Information Processing Systems 36, 24841–24858

Show all 77 references
  1. [9]

    Gpavatar: Generalizable and precise head avatar from image(s), in: The Twelfth International Conference on Learning Representations

    Chu,X.,Li,Y.,Zeng,A.,Yang,T.,Lin,L.,Liu,Y.,Harada,T.,2024. Gpavatar: Generalizable and precise head avatar from image(s), in: The Twelfth International Conference on Learning Representations. URL:https://openreview.net/forum?id=hgehGq2bDv

    2024

  2. [10]

    Hyperrealistic neural decoding for reconstructing faces from fmri activations via the gan latent space

    Dado, T., Güçlütürk, Y., Ambrogioni, L., Ras, G., Bosch, S., van Gerven, M., Güçlü, U., 2022. Hyperrealistic neural decoding for reconstructing faces from fmri activations via the gan latent space. Scientific reports 12, 141

    2022

  3. [11]

    Localizing brain function based on full multivariate activity patterns: The case of visual perception and emotion decoding

    David, I., Barrios, F., 2021. Localizing brain function based on full multivariate activity patterns: The case of visual perception and emotion decoding. bioRxiv , 2021–04

    2021

  4. [12]

    Arcface: Additive angular margin loss for deep face recognition, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp

    Deng, J., Guo, J., Xue, N., Zafeiriou, S., 2019. Arcface: Additive angular margin loss for deep face recognition, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690–4699

    2019

  5. [13]

    Portrait4d: Learning one-shot 4d head avatar synthesis using synthetic data, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Deng, Y., Wang, D., Ren, X., Chen, X., Wang, B., 2024a. Portrait4d: Learning one-shot 4d head avatar synthesis using synthetic data, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7119–7130

  6. [14]

    Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer

    Deng, Y., Wang, D., Wang, B., 2024b. Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer. arXiv preprint arXiv:2403.13570

    arXiv

  7. [15]

    andGoncalves, M., DuPre, E., Sitek, K.R., Gomez, D.E.P., Lurie, D.J., Ye, Z., Poldrack, R.A., Gorgolewski, K.J., 2018

    Esteban, O., Blair, R., Markiewicz, C.J., Berleant, S.L., Moodie, C., Ma, F., Isik, A.I., Erramuzpe, A., Kent, James D. andGoncalves, M., DuPre, E., Sitek, K.R., Gomez, D.E.P., Lurie, D.J., Ye, Z., Poldrack, R.A., Gorgolewski, K.J., 2018. fmriprep. Software doi:10.5281/ zenodo.852659

    2018

  8. [16]

    fMRIPrep: a robust preprocessing pipeline for functional MRI

    Esteban, O., Markiewicz, C., Blair, R.W., Moodie, C., Isik, A.I., ErramuzpeAliaga,A.,Kent,J.,Goncalves,M.,DuPre,E.,Snyder,M., Oya,H.,Ghosh,S.,Wright,J.,Durnez,J.,Poldrack,R.,Gorgolewski, K.J., 2019. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nature Methods 1...

    2019 doi

  9. [17]

    Alleviating the semantic gap for generalized fmri-to-image reconstruction

    Fang, T., Zheng, Q., Pan, G., 2023. Alleviating the semantic gap for generalized fmri-to-image reconstruction. Advances in Neural Information Processing Systems 36, 15096–15107

    2023

  10. [18]

    Learning an animatable detailed 3d face model from in-the-wild images

    Feng, Y., Feng, H., Black, M.J., Bolkart, T., 2021. Learning an animatable detailed 3d face model from in-the-wild images. ACM Transactions on Graphics (ToG) 40, 1–13

    2021

  11. [19]

    Ferrante, M., Ozcelik, F., Boccato, T., VanRullen, R., Toschi, N.,

  12. [20]

    Mind-3d: Reconstruct high-quality 3d objects in human brain

    Gao, J., Fu, Y., Wang, Y., Qian, X., Feng, J., Fu, Y., 2023. Mind-3d: Reconstruct high-quality 3d objects in human brain. arXiv preprint arXiv:2312.07485

    2023 arXiv

  13. [21]

    Gao, J., Liu, Y., Yang, B., Feng, J., Fu, Y., 2026. Cinebrain: A large- scale multi-modal audiovisual brain dataset for brain-conditioned video generation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 36224–36234

    2026

  14. [22]

    Neuroclips: Towards high- fidelityandsmoothfmri-to-videoreconstruction

    Gong, Z., Bao, G., Zhang, Q., Wan, Z., Miao, D., Wang, S., Zhu, L., Wang, C., Xu, R., Hu, L., et al., 2024. Neuroclips: Towards high- fidelityandsmoothfmri-to-videoreconstruction. AdvancesinNeural Information Processing Systems 37, 51655–51683

    2024

  15. [23]

    A large-scalefmridatasetforthevisualprocessingofnaturalisticscenes

    Gong, Z., Zhou, M., Dai, Y., Wen, Y., Liu, Y., Zhen, Z., 2023. A large-scalefmridatasetforthevisualprocessingofnaturalisticscenes. Scientific Data 10, 559

    2023

  16. [24]

    Neuro- gen:activationoptimizedimagesynthesisfordiscoveryneuroscience

    Gu, Z., Jamison, K.W., Khosla, M., Allen, E.J., Wu, Y., St-Yves, G., Naselaris, T., Kay, K., Sabuncu, M.R., Kuceyeski, A., 2022. Neuro- gen:activationoptimizedimagesynthesisfordiscoveryneuroscience. NeuroImage 247, 118812

    2022

  17. [25]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., Dai, B., 2024. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations . Page 13 of 22 fMRI2Face

    2024

  18. [26]

    Han, Y., Zhu, J., He, K., Chen, X., Ge, Y., Li, W., Li, X., Zhang, J., Wang,C.,Liu,Y.,2024.Face-adapterforpre-traineddiffusionmodels withfine-grainedidandattributecontrol,in:EuropeanConferenceon Computer Vision, Springer. pp. 20–36

    2024

  19. [27]

    Hebart, M.N., Contier, O., Teichmann, L., Rockter, A.H., Zheng, C.Y., Kidder, A., Corriveau, A., Vaziri-Pashkam, M., Baker, C.I.,

  20. [28]

    Video diffusion models

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J., 2022. Video diffusion models. Advances in neural information processing systems 35, 8633–8646

    2022

  21. [29]

    eLife 12, e82580

    Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behav- ior. eLife 12, e82580. URL:https://doi.org/10.7554/eLife.82580, doi:10.7554/eLife.82580

    doi

  22. [30]

    Huo, J., Wang, Y., Wang, Y., Qian, X., Li, C., Fu, Y., Feng, J.,

  23. [31]

    Generic decoding of seen and imagined objects using hierarchical visual features

    Horikawa, T., Kamitani, Y., 2017. Generic decoding of seen and imagined objects using hierarchical visual features. Nature commu- nications 8, 15037

    2017

  24. [32]

    Progressivegrowing of gans for improved quality, stability, and variation

    Karras,T.,Aila,T.,Laine,S.,Lehtinen,J.,2018. Progressivegrowing of gans for improved quality, stability, and variation. International Conference on Learning Representations

    2018

  25. [33]

    Avat3r: Large animatable gaussian reconstruction model for high-fidelity 3d head avatars, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Kirschstein, T., Romero, J., Sevastopolsky, A., Nießner, M., Saito, S., 2025. Avat3r: Large animatable gaussian reconstruction model for high-fidelity 3d head avatars, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12089–12100

    2025

  26. [34]

    Vace: All-in-one video creation and editing, in: Proceedings of the IEEE/CVFInternationalConferenceonComputerVision,pp.17191– 17202

    Jiang, Z., Han, Z., Mao, C., Zhang, J., Pan, Y., Liu, Y., 2025. Vace: All-in-one video creation and editing, in: Proceedings of the IEEE/CVFInternationalConferenceonComputerVision,pp.17191– 17202

    2025

  27. [35]

    Learning a model of facial shape and expression from 4d scans

    Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J., 2017. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph. 36. URL:https://doi.org/10.1145/3130800.3130813, doi:10. 1145/3130800.3130813

    2017

  28. [36]

    Visual image recon- structionfromhumanbrainactivityusingacombinationofmultiscale local image decoders

    Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.a., Morito, Y., Tanabe, H.C., Sadato, N., Kamitani, Y., 2008. Visual image recon- structionfromhumanbrainactivityusingacombinationofmultiscale local image decoders. Neuron 60, 915–929

    2008

  29. [37]

    Enhancing cross-subject fmri-to-video decoding with global-local functional alignment, in: European Conference on Computer Vision, Springer

    Li, C., Qian, X., Wang, Y., Huo, J., Xue, X., Fu, Y., Feng, J., 2024. Enhancing cross-subject fmri-to-video decoding with global-local functional alignment, in: European Conference on Computer Vision, Springer. pp. 353–369

    2024

  30. [38]

    Natural scene reconstruction from fmri signals using generative latent diffusion

    Ozcelik, F., VanRullen, R., 2023. Natural scene reconstruction from fmri signals using generative latent diffusion. Scientific Reports 13, 15666

    2023

  31. [39]

    Gaussianavatars:Photorealisticheadavatarswith rigged3dgaussians,in:ProceedingsoftheIEEE/CVFConferenceon Computer Vision and Pattern Recognition, pp

    Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., Nießner,M.,2024. Gaussianavatars:Photorealisticheadavatarswith rigged3dgaussians,in:ProceedingsoftheIEEE/CVFConferenceon Computer Vision and Pattern Recognition, pp. 20299–20309

    2024

  32. [40]

    Reconstructingnatural scenes from fmri patterns using bigbigan, in: 2020 International JointConferenceonNeuralNetworks,IJCNN2020,Glasgow,United Kingdom, July 19-24, 2020, IEEE

    Mozafari,M.,Reddy,L.,VanRullen,R.,2020. Reconstructingnatural scenes from fmri patterns using bigbigan, in: 2020 International JointConferenceonNeuralNetworks,IJCNN2020,Glasgow,United Kingdom, July 19-24, 2020, IEEE. pp. 1–8

    2020

  33. [41]

    fmri-pte: A large-scale fmri pretrained transformer encoder for multi-subject brain activity decoding

    Qian, X., Wang, Y., Huo, J., Feng, J., Fu, Y., 2023b. fmri-pte: A large-scale fmri pretrained transformer encoder for multi-subject brain activity decoding. arXiv preprint arXiv:2311.00342

    arXiv

  34. [42]

    Ren, Z., Li, J., Wu, L., Xue, X., Li, X., Yang, F., Jiao, Z., Gao, X.,

  35. [43]

    Joint fmridecodingandencodingwithlatentembeddingalignment

    Qian, X., Wang, Y., Fu, Y., Sun, X., Feng, J., Xue, X., 2023a. Joint fmridecodingandencodingwithlatentembeddingalignment. arXiv preprint arXiv:2303.14730

    arXiv

  36. [44]

    High-resolution image synthesis with latent diffusion models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Rombach,R.,Blattmann,A.,Lorenz,D.,Esser,P.,Ommer,B.,2022. High-resolution image synthesis with latent diffusion models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695

    2022

  37. [45]

    Linearreconstructionofperceivedimagesfromhumanbrainactivity

    Schoenmakers, S., Barth, M., Heskes, T., Van Gerven, M., 2013. Linearreconstructionofperceivedimagesfromhumanbrainactivity. NeuroImage 83, 951–961

    2013

  38. [46]

    Brain-drivenfacialimagereconstructionviastyleganinversion withimprovedidentityconsistency.PatternRecognition150,110331

  39. [47]

    Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning

    Ren, Z., Li, J., Xue, X., Li, X., Yang, F., Jiao, Z., Gao, X., 2021. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. NeuroImage 228, 117602

    2021

  40. [48]

    Deep imagereconstructionfromhumanbrainactivity

    Shen, G., Horikawa, T., Majima, K., Kamitani, Y., 2019. Deep imagereconstructionfromhumanbrainactivity. PLoScomputational biology 15, e1006633

    2019

  41. [49]

    Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N., 2019. First order motion model for image animation, in: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA

    2019

  42. [50]

    Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Demp- ster, A., Verlinde, N., Yundler, E., Weisberg, D., Norman, K., et al.,

  43. [51]

    Advances in Neural Information Processing Systems 36, 24705–24728

    Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems 36, 24705–24728

  44. [52]

    Mindeye2: Shared-subject models enable fmri- to-image with 1 hour of data

    Scotti,P.S.,Tripathy,M.,Torrico,C.,Kneeland,R.,Chen,T.,Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K.A., Abra- ham, T.M., 2024. Mindeye2: Shared-subject models enable fmri- to-image with 1 hour of data. International Conference on Machine Learning 235, 44038–44059

    2024

  45. [53]

    Reconstructing faces from fmri patterns using deep generative neural networks

    VanRullen, R., Reddy, L., 2019. Reconstructing faces from fmri patterns using deep generative neural networks. Communications biology 2, 193

    2019

  46. [54]

    Wan: Open and advanced large- scale video generative models

    Wan,T.,Wang,A.,Ai,B.,Wen,B.,Mao,C.,Xie,C.W.,Chen,D.,Yu, F., Zhao, H., Yang, J., et al., 2025. Wan: Open and advanced large- scale video generative models. arXiv preprint arXiv:2503.20314

    2025 arXiv

  47. [55]

    Motion representations for articulated animation, in: CVPR

    Siarohin, A., Woodford, O., Ren, J., Chai, M., Tulyakov, S., 2021. Motion representations for articulated animation, in: CVPR

    2021

  48. [56]

    Towardsaccurategenerativemodelsofvideo: A new metric & challenges

    Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michal- ski,M.,Gelly,S.,2018. Towardsaccurategenerativemodelsofvideo: A new metric & challenges. arXiv preprint arXiv:1812.01717

    2018 arXiv

  49. [57]

    Neuraldecoding with hierarchical generative models

    VanGerven,M.A.,DeLange,F.P.,Heskes,T.,2010. Neuraldecoding with hierarchical generative models. Neural computation 22, 3127– 3142

    2010

  50. [58]

    Image qualityassessment:fromerrorvisibilitytostructuralsimilarity

    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., 2004. Image qualityassessment:fromerrorvisibilitytostructuralsimilarity. IEEE transactions on image processing 13, 600–612

    2004

  51. [59]

    Motionctrl: A unified and flexible motion controller for videogeneration,in:ACMSIGGRAPH2024ConferencePapers,pp

    Wang,Z.,Yuan,Z.,Wang,X.,Li,Y.,Chen,T.,Xia,M.,Luo,P.,Shan, Y., 2024. Motionctrl: A unified and flexible motion controller for videogeneration,in:ACMSIGGRAPH2024ConferencePapers,pp. 1–11

    2024

  52. [60]

    Mead: A large-scale audio-visual dataset for emotional talking-face generation, in: European conference on computer vision, Springer

    Wang, K., Wu, Q., Song, L., Yang, Z., Wu, W., Qian, C., He, R., Qiao, Y., Loy, C.C., 2020. Mead: A large-scale audio-visual dataset for emotional talking-face generation, in: European conference on computer vision, Springer. pp. 700–717

    2020

  53. [61]

    One-shot free-view neural talking-head synthesis for video conferencing, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Wang, T.C., Mallya, A., Liu, M.Y., 2021. One-shot free-view neural talking-head synthesis for video conferencing, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10034–10044. doi:10.1109/CVPR46437.2021.00991

    2021

  54. [62]

    Visual lexicon:Richimagefeaturesinlanguagespace,in:Proceedingsofthe Computer Vision and Pattern Recognition Conference, pp

    Wang, X., Zhou, X., Fathi, A., Darrell, T., Schmid, C., 2025. Visual lexicon:Richimagefeaturesinlanguagespace,in:Proceedingsofthe Computer Vision and Pattern Recognition Conference, pp. 19736– 19747

    2025

  55. [63]

    Controllable mind visual diffusion model, in: Proceedings of the AAAI conference on artificial intelligence, pp

    Zeng, B., Li, S., Liu, X., Gao, S., Jiang, X., Tang, X., Hu, Y., Liu, J., Zhang, B., 2024. Controllable mind visual diffusion model, in: Proceedings of the AAAI conference on artificial intelligence, pp. 6935–6943

    2024

  56. [64]

    Face animation with an attribute-guided diffusion model, in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp

    Zeng, B., Liu, X., Gao, S., Liu, B., Li, H., Liu, J., Zhang, B., 2023. Face animation with an attribute-guided diffusion model, in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 628–637. doi:10.1109/CVPRW59228.2023. 00070

    2023

  57. [65]

    Neural encodinganddecodingwithdeeplearningfordynamicnaturalvision

    Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z., 2018. Neural encodinganddecodingwithdeeplearningfordynamicnaturalvision. Cerebral cortex 28, 4136–4160

    2018

  58. [66]

    Hunyuanportrait: Implicit condition control for enhanced portrait animation, in: Proceedings of theComputerVisionandPatternRecognitionConference,pp.15909– 15919

    Xu, Z., Yu, Z., Zhou, Z., Zhou, J., Jin, X., Hong, F.T., Ji, X., Zhu, J., Cai, C., Tang, S., et al., 2025. Hunyuanportrait: Implicit condition control for enhanced portrait animation, in: Proceedings of theComputerVisionandPatternRecognitionConference,pp.15909– 15919

    2025

  59. [67]

    Cogvideox: Text-to- video diffusion models with an expert transformer, in: International Page 14 of 22 fMRI2Face Conference on Learning Representations, pp

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al., 2025. Cogvideox: Text-to- video diffusion models with an expert transformer, in: International Page 14 of 22 fMRI2Face Conference on Learning Representations, pp. 83048–83077

    2025

  60. [70]

    The unreasonableeffectivenessofdeepfeaturesasaperceptualmetric,in: CVPR

    Zhang,R.,Isola,P.,Efros,A.A.,Shechtman,E.,Wang,O.,2018. The unreasonableeffectivenessofdeepfeaturesasaperceptualmetric,in: CVPR

    2018

  61. [71]

    Zhao, H., Zhang, J., Lai, Y.K., Zheng, Z., Xie, Y., Liu, Y., Li, K.,

  62. [73]

    More Details Weprovideadditionaldetailsonthedigitalhumancom- position and annotations in the fMRI-Face dataset

    Dataset 1.1. More Details Weprovideadditionaldetailsonthedigitalhumancom- position and annotations in the fMRI-Face dataset. In total, fMRI-Face contains 2,174 digital human videos, including 1,193 female and 981 male samples. Among them, 1,694 videosfeatureartisticdigitalhuma...

  63. [74]

    Brain-derived Appearance Context We provide the implementation details of the Appear- ance Context Predictorapp

    More Architecture Details 2.1. Brain-derived Appearance Context We provide the implementation details of the Appear- ance Context Predictorapp. Given an fMRI input window 𝐅∈ℝ 𝑇×𝑑 with𝑇= 5and𝑑= 8,921, we first project each fMRI measurement into the context dimension of the vid...

  64. [75]

    Aux. only

    Experiments 3.1. More Implementation Details Weprovideadditionaltrainingdetailsforthethreestages of fMRI2Face. All video diffusion experiments are con- ducted at a resolution of480×832with 24 frames. Brain-derivedAppearanceContext.WeuseWan2.1-T2V- 14Basthepretrainedvideodiffus...

    2007

  65. [76]

    Limitations and Future Work Although fMRI2Face achieves high-fidelity facial video reconstruction,severallimitationsremain.First,reconstruct- ing fine-grained transient facial motions remains challeng- ing. Although the proposed geometry-guided design im- proves facial structu...

  66. [77]

    First, they provide a new benchmark for dynamic face decoding from brain activity

    Broader Impact The fMRI-Face dataset and fMRI2Face framework may benefit several research areas. First, they provide a new benchmark for dynamic face decoding from brain activity. Compared with prior face-decoding datasets that mainly focusonstaticorlesscontrollablestimuli,fMR...

  67. [2022]

    15883–15892

    High-fidelityhumanavatarsfromasinglergbcamera,in:2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15883–15892. doi:10.1109/CVPR52688.2022.01544. Page 15 of 22 fMRI2Face Supplementary Material

    2022

  68. [2023]

    arXiv preprint arXiv:2305.11560

    Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560

    arXiv

  69. [2024]

    Neuropictor: Refining fmri-to-image reconstruction via multi- individualpretrainingandmulti-levelmodulation,in:EuropeanCon- ference on Computer Vision, Springer. pp. 56–73

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.