Pith. sign in

REVIEW 4 major objections 5 minor 68 references

QuantFormer: Learning to Quantize for Neural Activity Forecasting in Mouse Visual Cortex

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read QuantFormer reframes neural activity forecasting as classification: a vector-quantized transformer predicts which of 32 signal codes comes next, and the paper reports state-of-the-art results on the Allen mouse visual cortex dataset.

desk verdict The classification gains are real but modest; the forecasting benchmark claim rests on a metric that makes the mean baseline undefined, so the headline does not stand. read the letter →

arxiv 2412.07264 v1 pith:LK2UK2XW submitted 2024-12-10 q-bio.NC cs.CVeess.IVeess.SP

Salvatore Calcagno , Isaak Kavasidis , Simone Palazzo , Marco Brondi , Luca Sità , Giacomo Turri , Daniela Giordano , Vladimir R. Kostic
show 3 more authors
Tommaso Fellin Massimiliano Pontil Concetto Spampinato
This is my paper · ORCID
classification q-bio.NCcs.CVeess.IVeess.SP
keywords neuralactivityforecastingvectorquantizationtwo-photoncalciumimagingtransformermousevisualcortexmaskedauto-encodingtimeseriesneuronembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QuantFormer claims that forecasting neural activity from two-photon calcium imaging is better posed as classification than regression: the model learns a compact vocabulary of quantized signal codes and predicts which code will appear next, instead of predicting a continuous fluorescence trace. The paper argues this reframing fits the sparsity of neuronal activations, where most of the signal is flat and only occasional peaks carry information. The model is pre-trained as a masked auto-encoder on unlabeled Allen dataset traces, then adapted with neuron-specific and stimulus-specific tokens so a single model can serve arbitrary neuron populations and generalize across mice and stimuli. The authors report that QuantFormer surpasses existing forecasting methods on the Allen benchmark and treat this as a step toward a foundation model for the mouse visual cortex and toward real-time closed-loop optogenetic experiments.

What carries the argument

The load-bearing mechanism is a vector-quantized transformer auto-encoder trained as a masked auto-encoder. Input fluorescence traces are cut into non-overlapping patches, linearly projected into tokens, and a fraction of tokens is replaced by [MASK]; the encoder maps the masked sequence to continuous latents, a quantization layer snaps each latent to the nearest of $K=32$ learnable embeddings, and the decoder reconstructs the full trace from the quantized codes. Downstream, learnable [NEURON] and [STIM] tokens are prepended so the model can condition on neuron identity and stimulus, and forecasting becomes a cross-entropy classification over codebook indices for masked future patches, with the pre-trained decoder turning predicted codes back into signals. The quantization layer is what converts a regression problem into classification and what forces the model to reuse a small set of prototypical patterns, which the paper argues is the key to capturing sparse activations.

What would settle it

Re-run the forecasting comparison on the same Allen containers with un-normalized responses and include a constant-mean baseline; Table A-4 already provides the result, showing the baseline winning on MSE, MAE, SMAPE, Pearson correlation and SSIM, so the forecasting superiority claim would fail under standard evaluation unless the normalization is defended as the right protocol.

Watch

Extended reading notes

Core claim

The central discovery is that vector quantization can convert neural response forecasting into a code-prediction task that handles sparse activations better than regression. QuantFormer first learns, through masked auto-encoding, to map patches of calcium traces to one of 32 learnable codebook embeddings; at forecasting time the model masks the response window, classifies each masked patch into a code, and feeds the predicted codes through the frozen decoder to produce a continuous forecast. With this scheme, the paper reports slightly better stimulus-response classification than competitors (about 77.4% accuracy on the selected Allen containers) and better forecasting metrics under its gradient-sum normalization, with generalization across stimulus categories and subjects.

Load-bearing premise

The load-bearing premise is that the gradient-sum normalization used for forecasting metrics is the correct evaluation protocol: the paper's Appendix Table A-4 shows that with standard un-normalized MSE, MAE, SMAPE, Pearson correlation and SSIM, a constant-mean baseline beats QuantFormer on every metric, and the main Table II omits that baseline.

Editorial extensions

If this is right

  • If the claim is right, one model can forecast activity for any number of neurons by adding a new [NEURON] token, avoiding per-neuron or fixed-population models.
  • Forecasting as code classification should generalize to other sparse neural signals, since the quantization acts as a built-in sparsity prior; the paper shows pre-training transfers across stimulus types and mice.
  • Closed-loop optogenetics becomes more feasible: predicted future activity in the response window can be produced from the baseline trace alone, the input available in real time.
  • The 32-code vocabulary gives a compact, interpretable intermediate representation, and the paper shows neuron embeddings cluster by response statistics such as activation frequency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the forecast is decoded from discrete codes by a frozen decoder, one could swap the decoder for one trained on deconvolved or spiking signals and keep the encoder's learned dynamics, a route the paper leaves implicit.
  • The neuron-token design suggests continual learning: previously unseen neurons could be added with fresh tokens and prompt-only training, which would be a direct test of whether the encoder stores general V1 dynamics.
  • The 32-code vocabulary invites a stronger causal test: if codes are real neural primitives, perturbing a single predicted code should change the decoded trace in a stereotyped way, which is checkable in the paper's own reconstruction framework.
  • The same architecture could be pointed at other sensory cortices or species by retraining tokens, since nothing in the quantization or masking machinery is mouse-specific.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces QuantFormer, a transformer-based model that uses vector quantization to forecast neural activity from two-photon calcium imaging. The method pre-trains a masked autoencoder with a quantization layer, then adapts the encoder to downstream tasks by prepending neuron- and stimulus-specific tokens. For forecasting, the model predicts quantized code indices for masked future patches and reconstructs the continuous signal with the frozen decoder. The authors evaluate on a subset of the Allen Visual Coding dataset (11 mice, four stimulus categories) and report that QuantFormer outperforms state-of-the-art time-series transformers and BrainLM in both response classification and response forecasting, claiming to set a new benchmark in forecasting mouse visual cortex activity.

Significance. If the forecasting claims were sound, the paper would offer a valuable contribution: framing sparse calcium-imaging forecasting as a classification problem over a learned codebook, and using neuron-specific tokens to scale to arbitrary neuronal populations, are both useful ideas. The paper also includes ablations, generalization experiments across subjects and stimuli, and an interpretability analysis, which are commendable. However, the central forecasting benchmark claim is not supported by the reported evidence. The main forecasting table relies on a non-standard gradient-sum normalization that the paper's own appendix shows reverses the ranking against a simple mean-signal baseline; under standard un-normalized regression metrics, QuantFormer underperforms the baseline on every metric. The classification improvement over Cross-former is small and likely not statistically robust. These issues undermine the abstract's claims of state-of-the-art performance and a new benchmark, leaving a modest classification gain as the most defensible result.

major comments (4)
  1. [Sec. IV.C, Table II, Appendix C, Table A-4] The headline forecasting result is not supported because it is evaluated under a non-standard gradient-sum normalization that inverts the ranking relative to a constant-mean baseline. Table A-4 shows that under standard un-normalized metrics, the mean-signal baseline outperforms QuantFormer on every reported metric: MSE 0.095 vs 0.445, MAE 0.058 vs 0.236, SMAPE 0.829 vs 1.55, correlation 0.335 vs 0.138, and SSIM 0.122 vs 0.015. Appendix C states that the normalization was introduced because this baseline performs best on un-normalized metrics and that it assigns infinite MSE/MAE to a mean predictor; this baseline is omitted from Table II. Consequently, the claim that QuantFormer 'sets a new benchmark' in forecasting is an artifact of the chosen evaluation rule rather than evidence of superior forecasting quality.
  2. [Appendix C] The gradient-sum normalization has no independent justification as a forecasting metric for calcium imaging. Dividing each signal by the sum of the absolute values of its derivatives penalizes smooth, near-constant predictions, but two-photon fluorescence traces are predominantly flat with sparse transients. A metric that deliberately assigns an infinite error to a constant-mean forecaster is not a standard measure of time-series forecasting accuracy, and no external validation or precedent is given. Because the entire forecasting comparison in Table II depends on this metric, the main forecasting conclusion is not established.
  3. [Table I] The classification superiority over Cross-former is marginal and likely not statistically meaningful. QuantFormer reports accuracy 77.39 ± 3.88 vs 75.51 ± 4.45 and F1 66.94 ± 6.51 vs 63.89 ± 7.59, with overlapping standard deviations. The manuscript does not report a direct significance test between QuantFormer and Cross-former; the star markers in Table I indicate significance relative to a baseline, not between the two top methods. Therefore the abstract's claim of outperforming state-of-the-art methods in response classification is not convincingly supported.
  4. [Sec. III.D.2] The forecasting targets are codebook indices produced by the model's own pre-trained encoder, and the forecasted signal is obtained by decoding those indices with the frozen decoder. The paper does not report the reconstruction fidelity of the pre-trained autoencoder on the response window. Without this measurement, the reported forecasting errors conflate the downstream classifier's code-prediction error with the encoder-decoder reconstruction error, making it impossible to interpret which component limits forecasting performance. This missing analysis weakens the forecasting results even under the paper's chosen metric.
minor comments (5)
  1. [Sec. IV.B, Table A-2] The statement that performance decreases when diverging from K=32 and d=128 is contradicted by Table A-2, where K=16 yields lower MSE (0.015 vs 0.026) and comparable SSIM (0.091 vs 0.090) than K=32. The authors state that hyperparameters were selected by Pearson correlation, so the categorical claim of monotonically decreasing performance is inaccurate.
  2. [Table A-4] The 'Baseline' row is not clearly defined. The main text identifies an LSTM baseline that 'mostly predict[s] the signal's mean,' but LSTM is listed separately in the same table. Please specify whether the Baseline is a constant-mean predictor and how it differs from the LSTM baseline.
  3. [Abstract and Code Availability] The paper says 'Source code available online' but does not provide a URL or repository identifier. Please include the link in the manuscript.
  4. [Sec. IV.D] There is a typo in the first paragraph: 'undestrand' should be 'understand.'
  5. [Fig. 3 and Appendix D] The qualitative forecasting examples do not state whether they were randomly selected or chosen to illustrate the method's strengths. A selection criterion would help the reader assess the generality of the qualitative comparison.

Circularity Check

1 steps flagged · score 6.0 of 10

Forecasting benchmark is partly constructed: Appendix C adopts a derivative-sum normalization only after a mean-signal baseline beats QuantFormer on every standard metric, and that normalization by definition assigns infinite error to constant predictors, removing the baseline from Table II.

  1. self definitional [Section IV-C and Appendix C, Tables II and A-4]
    "Table A-4 presents forecasting metrics without normalization, where a basic mean signal baseline yields among the highest performance. However, regression metrics on un-normalized signals, given their sparse nature, does not accurately reflect the true forecasting capabilities of tested models. This motivates our normalization method, which normalizes signals dividing them by the sum of their absolute derivatives, emphasizing the rate of change."

    The main forecasting table uses a normalization adopted because the mean-signal baseline won on standard metrics. Dividing by the sum of absolute derivatives makes constant predictions undefined, so the baseline receives infinite MSE/MAE; the paper says this 'ensures that mean-baseline performance sets the lowest boundary.' Section IV-B reports the LSTM baseline 'mostly predict[s] the signal's mean,' and Table A-4 shows the baseline beats QuantFormer on every un-normalized metric (MSE 0.095 vs 0.445; Corr 0.335 vs 0.138; SSIM 0.122 vs 0.015). Table II omits the baseline and reports only normalized metrics. The 'new benchmark' claim thus follows from the metric definition rather than from independent evidence.

full rationale

The central classification result is a genuine empirical comparison: QuantFormer's accuracy and F1 are modestly better than Cross-former under standard balanced accuracy, precision, recall, and F1, so that part of the paper is self-contained. The forecasting targets are also not circular in the strict sense: although the downstream classifier is trained to predict code indices produced by the model's own frozen pre-trained encoder, the final evaluation compares decoder reconstructions against ground-truth fluorescence, and this is standard self-supervised representation learning rather than a derivation that equals its input. The material circularity is confined to the forecasting benchmark. Appendix C admits that a basic mean-signal baseline 'yields among the highest performance' on un-normalized metrics, and the normalization is introduced because of that observation. Mathematically, dividing by the sum of absolute derivatives makes constant predictors undefined, and the paper states this deliberately sets the mean baseline to the lowest boundary. Table A-4 shows the baseline winning on every standard regression metric, while main Table II omits the baseline and reports only the normalized metric on which QuantFormer wins. The claim of a new forecasting benchmark therefore depends on an evaluation rule that was selected after seeing the baseline's performance and defined to penalize exactly that baseline; this is a fitted criterion renamed as 'true forecasting capabilities.' The classification comparison and the architectural contributions retain independent content, so the paper is not entirely circular, but the central forecasting claim is partly constructed by the metric choice.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. The learnable neuron/stimulus tokens and codebook are model parameters, not scientific entities. The free parameters and axioms above are the main dependencies of the reported results.

free parameters (3)
  • Sparsity compensation weight beta = Set based on ratio of inactive to active neuron observations (exact value not reported)
    Eq. (2) introduces beta to reweight the reconstruction loss to compensate for rare activations; the value is derived from the training set's class imbalance, so it is data-dependent.
  • Codebook size K = 32
    Selected by hyperparameter search on drifting gratings (Appendix Table A-2) using downstream correlation; all main results use K=32.
  • Embedding/token dimensionality d = 128
    Selected jointly with K by hyperparameter search on drifting gratings; d=128 yields best correlation.
assumptions (4)
  • domain assumption The Allen Brain Observatory raw fluorescence traces are a valid target for forecasting neural activity.
    The paper argues raw fluorescence is more suitable for real-time optogenetics than deconvolved spikes (Sec. II), but this is a modeling choice, not a proven ground truth.
  • domain assumption A neuron is labeled active if mean fluorescence in the response window exceeds baseline by 10% (Chen et al. criterion).
    Used to define the classification labels a in Sec. III-B; inherited from [21].
  • ad hoc to paper The gradient-sum normalization (dividing signals by accumulated absolute derivatives) is an appropriate metric for forecasting quality.
    Introduced in Appendix C; the paper's own SOTA forecasting claims depend on this normalization, and without it a mean baseline outperforms the model.
  • domain assumption The masked-autoencoding pre-training with VQ yields a latent representation that transfers to forecasting and classification.
    Central modeling assumption of Sec. III-C; no formal guarantee, only empirical support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QuantFormer: Learning to Quantize for Neural Activity Forecasting in Mouse Visual Cortex." pith.science (2026). https://pith.science/paper/LK2UK2XW

@misc{pith2026241207264,
  author       = {Pith},
  title        = {Pith review of: QuantFormer: Learning to Quantize for Neural Activity Forecasting in Mouse Visual Cortex},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LK2UK2XW}},
  note         = {Machine review of arXiv:2412.07264}
}
read the original abstract

Understanding complex animal behaviors hinges on deciphering the neural activity patterns within brain circuits, making the ability to forecast neural activity crucial for developing predictive models of brain dynamics. This capability holds immense value for neuroscience, particularly in applications such as real-time optogenetic interventions. While traditional encoding and decoding methods have been used to map external variables to neural activity and vice versa, they focus on interpreting past data. In contrast, neural forecasting aims to predict future neural activity, presenting a unique and challenging task due to the spatiotemporal sparsity and complex dependencies of neural signals. Existing transformer-based forecasting methods, while effective in many domains, struggle to capture the distinctiveness of neural signals characterized by spatiotemporal sparsity and intricate dependencies. To address this challenge, we here introduce QuantFormer, a transformer-based model specifically designed for forecasting neural activity from two-photon calcium imaging data. Unlike conventional regression-based approaches, QuantFormerreframes the forecasting task as a classification problem via dynamic signal quantization, enabling more effective learning of sparse neural activation patterns. Additionally, QuantFormer tackles the challenge of analyzing multivariate signals from an arbitrary number of neurons by incorporating neuron-specific tokens, allowing scalability across diverse neuronal populations. Trained with unsupervised quantization on the Allen dataset, QuantFormer sets a new benchmark in forecasting mouse visual cortex activity. It demonstrates robust performance and generalization across various stimuli and individuals, paving the way for a foundational model in neural signal prediction.

Figures

Figures reproduced from arXiv: 2412.07264 by the authors.

Figure 1
Figure 1. Comparison of encoding, decoding, and forecasting tasks. Encoding methods take a stimulus and behavioral variables at time t to predict neural spikes at the same time point. In contrast, decoding methods work do the opposite, using spike responses at time t to predict behavioral variables for that time step. Neural forecasting differs from both, as it uses the stimulus at time t and raw fluorescence traces at time t… view at source ↗
Figure 2
Figure 2. QuantFormer architecture. During pre-training we employ a self-supervision quantization strategy that learns to reconstruct the randomly-masked patches along a quantization scheme. For response forecasting, [NEURON] and [STIM] tokens are prepended to the input, and neuronal response patches are masked; the model predicts for the masked patches quantized codes that are converted, through the quantization decoder lear… view at source ↗
Figure 3
Figure 3. Qualitative analysis of stimuli response forecasting performance by QuantFormer and its competitors: forecasting examples for each type of stimuli: drifting gratings (top-left), static gratings (top-right), natural scenes (bottom-left) and locally sparse noise (bottom-right). More examples can be found in Section D of the Appendix. TABLE III ABLATION STUDY FOR LEARNABLE TOKENS AND QUANTIZATION ON “DRIFTING GRATINGS”… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Attention maps for all stimulus types. Each row corresponds to a predicted code, while columns represent the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Interpretability of codes. (a) t-SNE of a codebook, with patterns representation in the same scale. We can appreciate along the first axis the amplitude [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Interpretability of neuron embeddings. We show t-SNE examples of neuron embeddings. We found that similar neurons in the latent space have also [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 57 canonical work pages

  1. [1]

    From the neuron doctrine to neural networks,

    R. Yuste, “From the neuron doctrine to neural networks,” Nature Reviews Neuroscience, vol. 16, no. 8, pp. 487–497, 2015. 1

    work page 2015

  2. [2]

    Neuronal ensembles: Building blocks of neural circuits,

    R. Yuste, R. Cossart, and E. Yaksi, “Neuronal ensembles: Building blocks of neural circuits,” Neuron, 2024. 1

    work page 2024

  3. [3]

    Functional imaging with cellular resolution reveals precise micro-architecture in visual cortex,

    K. Ohki, S. Chung, Y . H. Ch’ng, P. Kara, and R. C. Reid, “Functional imaging with cellular resolution reveals precise micro-architecture in visual cortex,” Nature, vol. 433, no. 7026, pp. 597–603, 2005. 1

    work page 2005

  4. [4]

    Highly ordered arrangement of single neurons in orientation pinwheels,

    K. Ohki, S. Chung, P. Kara, M. Hübener, T. Bonhoeffer, and R. C. Reid, “Highly ordered arrangement of single neurons in orientation pinwheels,” Nature, vol. 442, no. 7105, pp. 925–928, 2006. 1

    work page 2006

  5. [5]

    Scale-invariant movement encoding in the human motor system,

    N. K. Harpaz, T. Flash, and I. Dinstein, “Scale-invariant movement encoding in the human motor system,” Neuron, vol. 81, no. 2, pp. 452– 462, 2014. 1

    work page 2014

  6. [6]

    Context- dependent limb movement encoding in neuronal populations of motor cortex,

    W. Omlor, A.-S. Wahl, P. Sipilä, H. Lütcke, B. Laurenczy, I.-W. Chen, L. T. Sumanovski, M. van’t Hoff, P. Bethge, F. F. V oigt et al., “Context- dependent limb movement encoding in neuronal populations of motor cortex,” Nature communications, vol. 10, no. 1, p. 4812, 2019. 1

    work page 2019

  7. [7]

    Corticostriatal dynamics encode the refinement of specific behavioral variability during skill learning,

    F. J. Santos, R. F. Oliveira, X. Jin, and R. M. Costa, “Corticostriatal dynamics encode the refinement of specific behavioral variability during skill learning,” Elife, vol. 4, p. e09423, 2015. 1

    work page 2015

  8. [8]

    Brain-wide neural activity underlying memory-guided movement,

    S. Chen, Y . Liu, Z. A. Wang, J. Colonell, L. D. Liu, H. Hou, N.-W. Tien, T. Wang, T. Harris, S. Druckmann et al. , “Brain-wide neural activity underlying memory-guided movement,” Cell, vol. 187, no. 3, pp. 676–691,

    work page

Show all 68 references
  1. [9]

    Neural algorithms and circuits for motor planning,

    H. K. Inagaki, S. Chen, K. Daie, A. Finkelstein, L. Fontolan, S. Romani, and K. Svoboda, “Neural algorithms and circuits for motor planning,” Annual review of neuroscience , vol. 45, pp. 249–271, 2022. 1

    2022

  2. [10]

    The structures and functions of correlations in neural population codes,

    S. Panzeri, M. Moroni, H. Safaai, and C. D. Harvey, “The structures and functions of correlations in neural population codes,” Nature Reviews Neuroscience, vol. 23, no. 9, pp. 551–567, 2022. 1

    2022

  3. [11]

    Transition from predictable to variable motor cortex and striatal ensemble patterning during behavioral exploration,

    S. Kondapavulur, S. M. Lemke, D. Darevsky, L. Guo, P. Khanna, and K. Ganguly, “Transition from predictable to variable motor cortex and striatal ensemble patterning during behavioral exploration,” Nature communications, vol. 13, no. 1, p. 2450, 2022. 1

    2022

  4. [12]

    Visual stimuli recruit intrinsically generated cortical ensembles,

    J.-e. K. Miller, I. Ayzenshtat, L. Carrillo-Reid, and R. Yuste, “Visual stimuli recruit intrinsically generated cortical ensembles,” Proceedings of the National Academy of Sciences , vol. 111, no. 38, pp. E4053–E4061,

  5. [13]

    Self-healing codes: How stable neural populations can track continually reconfiguring neural representations,

    M. E. Rule and T. O’Leary, “Self-healing codes: How stable neural populations can track continually reconfiguring neural representations,” Proceedings of the National Academy of Sciences , vol. 119, no. 7, p. e2106692119, 2022. 1

    2022

  6. [14]

    Baseline brain activity fluctuations predict somatosensory perception in humans,

    M. Boly, E. Balteau, C. Schnakers, C. Degueldre, G. Moonen, A. Luxen, C. Phillips, P. Peigneux, P. Maquet, and S. Laureys, “Baseline brain activity fluctuations predict somatosensory perception in humans,” Pro- ceedings of the National Academy of Sciences , vol. 104, no. 29, p...

    2007

  7. [15]

    A sensorimotor circuit in mouse cortex for visual flow predictions,

    M. Leinweber, D. R. Ward, J. M. Sobczak, A. Attinger, and G. B. Keller, “A sensorimotor circuit in mouse cortex for visual flow predictions,” Neuron, vol. 95, no. 6, pp. 1420–1432, 2017. 1

    2017

  8. [16]

    Neurons learn by predicting future activity,

    A. Luczak, B. L. McNaughton, and Y . Kubo, “Neurons learn by predicting future activity,” Nature machine intelligence , vol. 4, no. 1, pp. 62–72,

  9. [17]

    Fully integrated silicon probes for high-density recording of neural activity,

    J. J. Jun, N. A. Steinmetz, J. H. Siegle, D. J. Denman, M. Bauza, B. Barbarits, A. K. Lee, C. A. Anastassiou, A. Andrei, Ç. Aydın et al., “Fully integrated silicon probes for high-density recording of neural activity,” Nature, vol. 551, no. 7679, pp. 232–236, 2017. 1

    2017

  10. [18]

    Distributed coding of choice, action and engagement across the mouse brain,

    N. A. Steinmetz, P. Zatka-Haas, M. Carandini, and K. D. Harris, “Distributed coding of choice, action and engagement across the mouse brain,” Nature, vol. 576, no. 7786, pp. 266–273, 2019. 1

    2019

  11. [19]

    Two-photon laser scanning fluorescence microscopy,

    W. Denk, J. H. Strickler, and W. W. Webb, “Two-photon laser scanning fluorescence microscopy,” Science, vol. 248, no. 4951, pp. 73–76, 1990. 1

    1990

  12. [20]

    Deep tissue two-photon microscopy,

    F. Helmchen and W. Denk, “Deep tissue two-photon microscopy,” Nature methods, vol. 2, no. 12, pp. 932–940, 2005. 1

    2005

  13. [21]

    Ultrasensitive fluorescent proteins for imaging neuronal activity,

    T.-W. Chen, T. J. Wardill, Y . Sun, S. R. Pulver, S. L. Renninger, A. Baohan, E. R. Schreiter, R. A. Kerr, M. B. Orger, V . Jayaraman et al., “Ultrasensitive fluorescent proteins for imaging neuronal activity,” Nature, vol. 499, no. 7458, pp. 295–300, 2013. 1, 3

    2013

  14. [22]

    High-performance calcium sensors for imaging activity in neuronal populations and microcompartments,

    H. Dana, Y . Sun, B. Mohar, B. K. Hulse, A. M. Kerlin, J. P. Hasseman, G. Tsegaye, A. Tsang, A. Wong, R. Patel et al. , “High-performance calcium sensors for imaging activity in neuronal populations and microcompartments,” Nature methods, vol. 16, no. 7, pp. 649–657, 2019. 1

    2019

  15. [23]

    Behaviorally relevant decision coding in primary somatosensory cortex neurons,

    C. Buetfering, Z. Zhang, M. Pitsiani, J. Smallridge, E. Boven, S. McEl- ligott, and M. Häusser, “Behaviorally relevant decision coding in primary somatosensory cortex neurons,” Nature neuroscience, vol. 25, no. 9, pp. 1225–1236, 2022. 1

    2022

  16. [24]

    Controlling visually guided behavior by holographic recalling of cortical ensembles,

    L. Carrillo-Reid, S. Han, W. Yang, A. Akrouh, and R. Yuste, “Controlling visually guided behavior by holographic recalling of cortical ensembles,” Cell, vol. 178, no. 2, pp. 447–457, 2019. 1

    2019

  17. [25]

    Task-specific modulation of corticospinal neuron activity during motor learning in mice,

    N. Serradj, F. Marino, Y . Moreno-López, A. Bernstein, S. Agger, M. Soliman, A. Sloan, and E. Hollis, “Task-specific modulation of corticospinal neuron activity during motor learning in mice,” Nature Communications, vol. 14, no. 1, p. 2708, 2023. 1

    2023

  18. [26]

    A unified, scalable framework for neural population decoding,

    M. Azabou, V . Arora, V . Ganesh, X. Mao, S. Nachimuthu, M. Mendelson, B. Richards, M. Perich, G. Lajoie, and E. L. Dyer, “A unified, scalable framework for neural population decoding,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. 1, 2

    2023

  19. [27]

    Neural data transformer 2: Multi-context pretraining for neural spiking activity,

    J. Ye, J. Collinger, L. Wehbe, and R. Gaunt, “Neural data transformer 2: Multi-context pretraining for neural spiking activity,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran As...

    2023

  20. [28]

    Neuro- former: Multimodal and multitask generative pretraining for brain data,

    A. Antoniades, Y . Yu, J. Canzano, W. Wang, and S. L. Smith, “Neuro- former: Multimodal and multitask generative pretraining for brain data,”

  21. [29]

    The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos,

    P. Turishcheva, P. G. Fahey, L. Hansel, R. Froebe, K. Ponder, M. Vystr ˇcilová, K. F. Willeke, M. Bashiri, E. Wang, Z. Ding, A. S. Tolias, F. H. Sinz, and A. S. Ecker, “The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos,”

  22. [30]

    Retrospective for the dynamic sensorium competition for predicting large-scale mouse primary visual cortex activity from videos,

    P. Turishcheva, P. G. Fahey, M. Vystr ˇcilová, L. Hansel, R. Froebe, K. Ponder, Y . Qiu, K. F. Willeke, M. Bashiri, R. Baikulov, Y . Zhu, L. Ma, S. Yu, T. Huang, B. M. Li, W. D. Wulf, N. Kudryashova, M. H. Hennig, N. L. Rochefort, A. Onken, E. Wang, Z. Ding, A. S. Tolias, F. H...

    2024 arXiv

  23. [31]

    V1t: large-scale mouse v1 response prediction using a vision transformer,

    B. M. Li, I. M. Cornacchia, N. Rochefort, and A. Onken, “V1t: large-scale mouse v1 response prediction using a vision transformer,” Transactions on Machine Learning Research , 2023. [Online]. Available: https://openreview.net/forum?id=qHZs2p4ZD4 1, 2

    2023

  24. [32]

    Multimodal deep learning model unveils behavioral dynamics of V1 activity in freely moving mice,

    A. Xu, Y . Hou, C. M. Niell, and M. Beyeler, “Multimodal deep learning model unveils behavioral dynamics of V1 activity in freely moving mice,” bioRxivorg, May 2023. 1, 2

    2023

  25. [33]

    Stimulus domain transfer in recurrent models for large scale cortical population prediction on video,

    F. Sinz, A. S. Ecker, P. Fahey, E. Walker, E. Cobos, E. Froudarakis, D. Yatsenko, Z. Pitkow, J. Reimer, and A. Tolias, “Stimulus domain transfer in recurrent models for large scale cortical population prediction on video,” in Advances in Neural Information Processing Systems ,...

    2018

  26. [34]

    Collective dynamics in human and monkey sensorimotor cortex: predicting single neuron spikes,

    W. Truccolo, L. Hochberg, and J. Donoghue, “Collective dynamics in human and monkey sensorimotor cortex: predicting single neuron spikes,” Nature Neuroscience , vol. 13, no. 1, pp. 105–111, 2010. [Online]. Available: https://doi.org/10.1038/nn.2455 1

    2010 doi

  27. [35]

    Brain-score: Which artificial neural network for object recognition is most brain-like?

    M. Schrimpf, J. Kubilius, H. Hong, N. J. Majaj, R. Rajalingham, E. B. Issa, K. Kar, P. Bashivan, J. Prescott-Roy, F. Geiger, K. Schmidt, D. L. K. Yamins, and J. J. DiCarlo, “Brain-score: Which artificial neural network for object recognition is most brain-like?” bioRxiv prepri...

    2018 doi

  28. [36]

    Neural latents benchmark ’21: Evaluating latent variable models of neural population activity,

    F. Pei, J. Ye, D. M. Zoltowski, A. Wu, R. H. Chowdhury, H. Sohn, J. E. O’Doherty, K. V . Shenoy, M. T. Kaufman, M. Churchland, M. Jazayeri, L. E. Miller, J. Pillow, I. M. Park, E. L. Dyer, and C. Pandarinath, “Neural latents benchmark ’21: Evaluating latent variable models of ...

  29. [37]

    Pixel recurrent neural networks,

    A. Van Den Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” in Proceedings of the International Conference on Machine Learning (ICML) , 2016. [Online]. Available: http://proceedings.mlr.press/v48/oord16.html 1

    2016

  30. [38]

    Wavenet: A generative model for raw audio,

    A. Van Den Oord, S. Dieleman, N. Zeghidour, F. Tacchino, S. Ganaie, G. Haffari, and A. Senior, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499 , 2016. [Online]. Available: https://arxiv.org/abs/1609.03499 1, 3 11

    2016 arXiv

  31. [39]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. 1, 2, 4

    2018 arXiv

  32. [40]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16 000–16 009. 1, 2, 3

    2022

  33. [41]

    A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex,

    S. E. de Vries, J. A. Lecoq, M. A. Buice, P. A. Groblewski, G. K. Ocker, M. Oliver, D. Feng, N. Cain, P. Ledochowitsch, D. Millman et al., “A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex,” Nature neuroscience, vol. 23...

    2020

  34. [42]

    Fast and robust active neuron segmentation in two-photon calcium imaging using spatiotemporal deep learning,

    S. Soltanian-Zadeh, K. Sahingur, S. Blau, Y . Gong, and S. Farsiu, “Fast and robust active neuron segmentation in two-photon calcium imaging using spatiotemporal deep learning,” Proceedings of the National Academy of Sciences , vol. 116, no. 17, pp. 8554–8563, 2019. 2

    2019

  35. [43]

    A deep-learning approach for online cell identification and trace extraction in functional two-photon calcium imaging,

    L. Sità, M. Brondi, P. Lagomarsino de Leon Roig, S. Curreli, M. Panniello, D. Vecchia, and T. Fellin, “A deep-learning approach for online cell identification and trace extraction in functional two-photon calcium imaging,” Nature Communications, vol. 13, no. 1, p. 1529, 2022. 2, 5

    2022

  36. [44]

    Segmentation of neurons from fluorescence calcium recordings beyond real time,

    Y . Bao, S. Soltanian-Zadeh, S. Farsiu, and Y . Gong, “Segmentation of neurons from fluorescence calcium recordings beyond real time,” Nature machine intelligence, vol. 3, no. 7, pp. 590–600, 2021. 2

    2021

  37. [45]

    Neuroseg-ii: A deep learning approach for generalized neuron segmentation in two-photon ca2+ imaging,

    Z. Xu, Y . Wu, J. Guan, S. Liang, J. Pan, M. Wang, Q. Hu, H. Jia, X. Chen, and X. Liao, “Neuroseg-ii: A deep learning approach for generalized neuron segmentation in two-photon ca2+ imaging,” Frontiers in Cellular Neuroscience, vol. 17, p. 1127847, 2023. 2

    2023

  38. [46]

    En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting,

    S. Li, X. Jin, Y . Xuan, X. Zhou, W. Chen, Y .-X. Wang, and X. Yan, “En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting,” Advances in neural information processing systems, vol. 32, 2019. 2

    2019

  39. [47]

    Informer: Beyond efficient transformer for long sequence time-series forecasting,

    H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, 2021, pp. 11 106–11 115. 2, 6, 14

    2021

  40. [48]

    A time series is worth 64 words: Long-term forecasting with transformers,

    Y . Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” in International Conference on Learning Representations , 2023. 2

    2023

  41. [49]

    Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,

    Y . Zhang and J. Yan, “Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,” in The eleventh international conference on learning representations , 2022. 2, 4, 6, 7, 14

    2022

  42. [50]

    Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting,

    S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. X. Liu, and S. Dustdar, “Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting,” in International conference on learning representations, 2021. 2

    2021

  43. [51]

    Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting,

    T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting,” in International conference on machine learning . PMLR, 2022, pp. 27 268–27 286. 2

    2022

  44. [52]

    Masked autoencoders that listen,

    P.-Y . Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that listen,” Advances in Neural Information Processing Systems , vol. 35, pp. 28 708–28 720, 2022. 2

    2022

  45. [53]

    Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,” Advances in neural information processing systems , vol. 35, pp. 10 078– 10 093, 2022. 2

    2022

  46. [54]

    Taming transformers for high- resolution image synthesis,

    P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high- resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 12 873–12 883. 2

    2021

  47. [55]

    Brainlm: A foundation model for brain activity recordings,

    J. Ortega Caro, A. H. Oliveira Fonseca, C. Averill, S. A. Rizvi, M. Rosati, J. L. Cross, P. Mittal, E. Zappala, D. Levine, R. M. Dhodapkar et al., “Brainlm: A foundation model for brain activity recordings,” bioRxiv, pp. 2023–09, 2023. 3, 6

    2023

  48. [56]

    Swift: Swin 4d fmri transformer,

    P. Y . Kim, J. Kwon, S. Joo, S. Bae, D. Lee, Y . Jung, S. Yoo, J. Cha, and T. Moon, “Swift: Swin 4d fmri transformer,” arXiv preprint arXiv:2307.05916, 2023. 3

    2023 arXiv

  49. [57]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022. 3

    2021

  50. [58]

    Relationship between simultaneously recorded spiking activity and fluorescence signal in gcamp6 transgenic mice,

    L. Huang, P. Ledochowitsch, U. Knoblich, J. Lecoq, G. J. Murphy, R. C. Reid, S. E. de Vries, C. Koch, H. Zeng, M. A. Buice, J. Waters, and L. Li, “Relationship between simultaneously recorded spiking activity and fluorescence signal in gcamp6 transgenic mice,” eLife, vol. 10, ...

    2021 doi

  51. [59]

    Towards a foundation model of the mouse visual cortex,

    E. Y . Wang, P. G. Fahey, K. Ponder, Z. Ding, A. Chang, T. Muhammad, S. Patel, Z. Ding, D. Tran, J. Fu et al., “Towards a foundation model of the mouse visual cortex,” bioRxiv, 2023. 3

    2023

  52. [60]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020. 3, 4

    2010 arXiv

  53. [61]

    Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks,

    M. Huh, B. Cheung, P. Agrawal, and P. Isola, “Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks,” in International Conference on Machine Learning . PMLR, 2023. 3, 4

    2023

  54. [62]

    Understanding straight-through estimator in training activation quantized neural nets,

    P. Yin, J. Lyu, S. Zhang, S. Osher, Y . Qi, and J. Xin, “Understanding straight-through estimator in training activation quantized neural nets,” arXiv preprint arXiv:1903.05662 , 2019. 3

    1903 arXiv

  55. [63]

    Technical whitepaper: Stimulus set and response analysis,

    A. B. Observatory, “Technical whitepaper: Stimulus set and response analysis,” 2017. [Online]. Available: https://community.brain-map.org/ uploads/short-url/uOe7nlLdLLIIivh5PeL8a0g7gV7.pdf 5

    2017

  56. [64]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014. 6

    2014 arXiv

  57. [65]

    Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting,

    H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting,” CoRR, vol. abs/2106.13008, 2021. [Online]. Available: https://arxiv.org/ abs/2106.13008 6, 7, 14

    2021 arXiv

  58. [66]

    The mouse genome database (mgd): facilitating mouse as a model for human biology,

    J. T. Eppig, J. A. Blake, C. J. Bult, J. A. Kadin, J. E. Richardson, and the Mouse Genome Database Group, “The mouse genome database (mgd): facilitating mouse as a model for human biology,” Nucleic Acids Research, vol. 43, no. D1, pp. D726–D736, 2015. 6 12 APPENDIX A. Dataset ...

    2015

  59. [2021]

    Available: https://arxiv.org/abs/2109.04463 1, 3

    [Online]. Available: https://arxiv.org/abs/2109.04463 1, 3

    arXiv

  60. [2024]

    Available: https://arxiv.org/abs/2305.19654 1, 2

    [Online]. Available: https://arxiv.org/abs/2305.19654 1, 2

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.