{"id":"35e1da96-9ae5-47d9-9e57-1df036a23cbd","arxiv_id":"2412.07264","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"QuantFormer forecasts neural activity by quantizing calcium signals into codes and classifying the future codes, but its headline forecasting gains over a trivial mean baseline disappear under standard error metrics.","lead":"A transformer model for brain signals, QuantFormer predicts future calcium-imaging activity by turning the task into a code-classification problem. It reports top results on mouse visual cortex data, but only under a specially chosen scoring rule that makes a simple 'predict the average' baseline look bad.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The forecasting benchmark claim is not supported: under standard un-normalized metrics the paper's own Table A-4 shows a constant-mean baseline beats QuantFormer on every regression metric, and the main Table II omits this baseline under a normalization that makes the baseline undefined.","rationale":"The reader's weakest assumption identifies the exact load-bearing issue in the paper's main claim. The strongest claim promises state-of-the-art forecasting, but the forecasting evidence is conditional on a normalization that was designed to penalize mean predictions. The paper's own appendix provides the counterexample: under conventional regression metrics, a constant-mean baseline is better than QuantFormer on every reported measure. Removing that baseline from the main table and reporting only the custom metric obscures the fact that the proposed model does not forecast the time series better than trivial predictors under standard error measures. I agree with the REJECT verdict because the primary evidence for the central claim fails. I also note secondary weaknesses that reinforce the rejection: the abstract says 'trained on the Allen dataset' and 'sets a new benchmark' while the conclusion acknowledges training on a subset; the claimed source code is not actually linked in the manuscript; and there is no independent verification such as released code or a parameter-free derivation. None of these secondary points are needed for the decision, but they further support not accepting the benchmark claim as stated.","tokens_in":19530,"tokens_out":2779,"duration_ms":30619,"concrete_test":"Re-run the forecasting experiments on the same 11 containers and stimuli, reporting a constant-mean baseline alongside QuantFormer under both (a) standard un-normalized MSE, MAE, SMAPE, Pearson correlation, and SSIM, and (b) the Appendix C gradient normalization. If under (a) the baseline again outperforms QuantFormer on all metrics, the forecasting benchmark claim should be downgraded to 'improves activation-shape capture under a custom metric' rather than 'new benchmark.' If under (a) QuantFormer actually wins, or if the gradient normalization is independently established as a standard protocol for this task, then the reader's concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that QuantFormer 'sets a new benchmark in forecasting mouse visual cortex activity' rests on Table II, which reports metrics after the Appendix C gradient normalization. Appendix C says this normalization was introduced because a mean-signal baseline performs best on un-normalized regression metrics, and Table A-4 confirms that on standard MSE, MAE, SMAPE, Pearson correlation, and SSIM, the baseline outperforms QuantFormer (e.g., MSE 0.095 vs 0.445, Corr 0.335 vs 0.138, SSIM 0.122 vs 0.015). The normalization divides by the sum of absolute derivatives of the prediction, so a constant-mean forecaster has denominator zero and is assigned infinite MSE/MAE, mechanically forcing it to 'set the lowest boundary.' No independent justification is given that this normalized metric is the accepted evaluation protocol for single-trial calcium-imaging forecasting. Consequently, the headline forecasting result appears to be an artifact of an evaluation rule chosen to invert the baseline ranking, rather than evidence of a new state of the art. The classification gain over Cross-former is modest and does not support the abstract's broader forecasting benchmark claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QuantFormer, a transformer-based model that uses vector quantization to forecast neural activity from two-photon calcium imaging. The method pre-trains a masked autoencoder with a quantization layer, then adapts the encoder to downstream tasks by prepending neuron- and stimulus-specific tokens. For forecasting, the model predicts quantized code indices for masked future patches and reconstructs the continuous signal with the frozen decoder. The authors evaluate on a subset of the Allen Visual Coding dataset (11 mice, four stimulus categories) and report that QuantFormer outperforms state-of-the-art time-series transformers and BrainLM in both response classification and response forecasting, claiming to set a new benchmark in forecasting mouse visual cortex activity.","tokens_in":19752,"tokens_out":5026,"duration_ms":50812,"significance":"If the forecasting claims were sound, the paper would offer a valuable contribution: framing sparse calcium-imaging forecasting as a classification problem over a learned codebook, and using neuron-specific tokens to scale to arbitrary neuronal populations, are both useful ideas. The paper also includes ablations, generalization experiments across subjects and stimuli, and an interpretability analysis, which are commendable. However, the central forecasting benchmark claim is not supported by the reported evidence. The main forecasting table relies on a non-standard gradient-sum normalization that the paper's own appendix shows reverses the ranking against a simple mean-signal baseline; under standard un-normalized regression metrics, QuantFormer underperforms the baseline on every metric. The classification improvement over Cross-former is small and likely not statistically robust. These issues undermine the abstract's claims of state-of-the-art performance and a new benchmark, leaving a modest classification gain as the most defensible result.","major_comments":[{"comment":"The headline forecasting result is not supported because it is evaluated under a non-standard gradient-sum normalization that inverts the ranking relative to a constant-mean baseline. Table A-4 shows that under standard un-normalized metrics, the mean-signal baseline outperforms QuantFormer on every reported metric: MSE 0.095 vs 0.445, MAE 0.058 vs 0.236, SMAPE 0.829 vs 1.55, correlation 0.335 vs 0.138, and SSIM 0.122 vs 0.015. Appendix C states that the normalization was introduced because this baseline performs best on un-normalized metrics and that it assigns infinite MSE/MAE to a mean predictor; this baseline is omitted from Table II. Consequently, the claim that QuantFormer 'sets a new benchmark' in forecasting is an artifact of the chosen evaluation rule rather than evidence of superior forecasting quality.","section":"Sec. IV.C, Table II, Appendix C, Table A-4"},{"comment":"The gradient-sum normalization has no independent justification as a forecasting metric for calcium imaging. Dividing each signal by the sum of the absolute values of its derivatives penalizes smooth, near-constant predictions, but two-photon fluorescence traces are predominantly flat with sparse transients. A metric that deliberately assigns an infinite error to a constant-mean forecaster is not a standard measure of time-series forecasting accuracy, and no external validation or precedent is given. Because the entire forecasting comparison in Table II depends on this metric, the main forecasting conclusion is not established.","section":"Appendix C"},{"comment":"The classification superiority over Cross-former is marginal and likely not statistically meaningful. QuantFormer reports accuracy 77.39 ± 3.88 vs 75.51 ± 4.45 and F1 66.94 ± 6.51 vs 63.89 ± 7.59, with overlapping standard deviations. The manuscript does not report a direct significance test between QuantFormer and Cross-former; the star markers in Table I indicate significance relative to a baseline, not between the two top methods. Therefore the abstract's claim of outperforming state-of-the-art methods in response classification is not convincingly supported.","section":"Table I"},{"comment":"The forecasting targets are codebook indices produced by the model's own pre-trained encoder, and the forecasted signal is obtained by decoding those indices with the frozen decoder. The paper does not report the reconstruction fidelity of the pre-trained autoencoder on the response window. Without this measurement, the reported forecasting errors conflate the downstream classifier's code-prediction error with the encoder-decoder reconstruction error, making it impossible to interpret which component limits forecasting performance. This missing analysis weakens the forecasting results even under the paper's chosen metric.","section":"Sec. III.D.2"}],"minor_comments":[{"comment":"The statement that performance decreases when diverging from K=32 and d=128 is contradicted by Table A-2, where K=16 yields lower MSE (0.015 vs 0.026) and comparable SSIM (0.091 vs 0.090) than K=32. The authors state that hyperparameters were selected by Pearson correlation, so the categorical claim of monotonically decreasing performance is inaccurate.","section":"Sec. IV.B, Table A-2"},{"comment":"The 'Baseline' row is not clearly defined. The main text identifies an LSTM baseline that 'mostly predict[s] the signal's mean,' but LSTM is listed separately in the same table. Please specify whether the Baseline is a constant-mean predictor and how it differs from the LSTM baseline.","section":"Table A-4"},{"comment":"The paper says 'Source code available online' but does not provide a URL or repository identifier. Please include the link in the manuscript.","section":"Abstract and Code Availability"},{"comment":"There is a typo in the first paragraph: 'undestrand' should be 'understand.'","section":"Sec. IV.D"},{"comment":"The qualitative forecasting examples do not state whether they were randomly selected or chosen to illustrate the method's strengths. A selection criterion would help the reader assess the generality of the qualitative comparison.","section":"Fig. 3 and Appendix D"}],"recommendation":"reject","confidential_remarks":"The referee agrees with the main thrust of the stress-test note: the forecasting benchmark claim rests on a normalization that is explicitly designed to invert the baseline ranking, and the paper's own Table A-4 shows that under standard metrics a simple mean baseline beats QuantFormer. This is a load-bearing flaw in the central contribution. The classification result is modest and not convincingly significant. If the authors were to resubmit with a reframed scope—dropping the forecasting benchmark claim, reporting standard metrics, and focusing on the classification setup with careful significance testing—a future submission could be viable, but the current manuscript should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The forecasting benchmark claim does not hold up. The paper's own Table A-4 shows that under un-normalized MSE, MAE, SMAPE, Pearson correlation, and SSIM, a constant-mean baseline beats QuantFormer on every measure (e.g., MSE 0.095 vs 0.445, Corr 0.335 vs 0.138). The main Table II then reports results under a normalization that divides each signal by the sum of absolute derivatives, which makes the mean baseline undefined (infinite error) and mechanically inverts the ranking. Appendix C states this was done deliberately to \"ensure that mean-baseline performance sets the lowest boundary.\" That is not a neutral evaluation choice; it is a rule chosen to make the main result look good. Omitting the baseline from Table II is a serious presentation problem.\n\nWhat is genuinely new here is the combination: vector-quantized masked autoencoding with neuron-specific tokens applied to single-trial calcium-imaging forecasting, framed as code classification. The classification results are modest but consistent – QuantFormer edges out Cross-former on accuracy and F1 across stimuli. The leave-one-out generalization experiments and the interpretability analysis are useful contributions. The method is a reasonable engineering contribution that could be useful for closed-loop optogenetics.\n\nThe soft spots beyond the metric issue: the forecasting targets are code indices produced by the model's own pre-trained encoder, so the classifier is learning to predict the model's internal representations rather than the fluorescence directly. The final evaluation is against ground truth, which mitigates the circularity, but it remains a concern. The abstract says \"trained on the Allen dataset\" when the experiments use 11 of 400+ containers; the conclusion correctly says \"a subset,\" so the abstract overstates. The \"source code available online\" claim is not backed by a link in this version.\n\nWho is this for? Researchers working on neural forecasting and closed-loop experiments might find the architecture worth exploring, and the paper raises a legitimate question about how to evaluate sparse neural signal forecasts. But as a benchmark claim, it fails. This deserves a serious referee because the problems are addressable – the authors could report un-normalized metrics with the mean baseline included, justify the normalization independently, and tone down the claims. A proper review process should catch this.\n\nMy recommendation: do not reject without review, but send to referees with emphasis on the evaluation protocol, or if the venue requires the headline claim to stand, reject as is. The paper needs major revision to be trustworthy.","headline":"The classification gains are real but modest; the forecasting benchmark claim rests on a metric that makes the mean baseline undefined, so the headline does not stand.","tokens_in":20321,"tokens_out":2846,"would_cite":false,"duration_ms":28941,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QuantFormer reframes neural activity forecasting as classification: a vector-quantized transformer predicts which of 32 signal codes comes next, and the paper reports state-of-the-art results on the Allen mouse visual cortex dataset.","keywords":["neural activity forecasting","vector quantization","two-photon calcium imaging","transformer","mouse visual cortex","masked auto-encoding","time series forecasting","neuron embeddings"],"falsifier":"Re-run the forecasting comparison on the same Allen containers with un-normalized responses and include a constant-mean baseline; Table A-4 already provides the result, showing the baseline winning on MSE, MAE, SMAPE, Pearson correlation and SSIM, so the forecasting superiority claim would fail under standard evaluation unless the normalization is defended as the right protocol.","tokens_in":19318,"feed_emoji":"🧠","tokens_out":6845,"duration_ms":69088,"temperature":0.7,"pith_summary":"QuantFormer claims that forecasting neural activity from two-photon calcium imaging is better posed as classification than regression: the model learns a compact vocabulary of quantized signal codes and predicts which code will appear next, instead of predicting a continuous fluorescence trace. The paper argues this reframing fits the sparsity of neuronal activations, where most of the signal is flat and only occasional peaks carry information. The model is pre-trained as a masked auto-encoder on unlabeled Allen dataset traces, then adapted with neuron-specific and stimulus-specific tokens so a single model can serve arbitrary neuron populations and generalize across mice and stimuli. The authors report that QuantFormer surpasses existing forecasting methods on the Allen benchmark and treat this as a step toward a foundation model for the mouse visual cortex and toward real-time closed-loop optogenetic experiments.","feed_headline":"Forecasting mouse brain activity as code classification","feed_subtitle":"A vector-quantized transformer predicts calcium traces from the Allen dataset, generalizing across stimuli and mice.","key_machinery":"The load-bearing mechanism is a vector-quantized transformer auto-encoder trained as a masked auto-encoder. Input fluorescence traces are cut into non-overlapping patches, linearly projected into tokens, and a fraction of tokens is replaced by [MASK]; the encoder maps the masked sequence to continuous latents, a quantization layer snaps each latent to the nearest of $K=32$ learnable embeddings, and the decoder reconstructs the full trace from the quantized codes. Downstream, learnable [NEURON] and [STIM] tokens are prepended so the model can condition on neuron identity and stimulus, and forecasting becomes a cross-entropy classification over codebook indices for masked future patches, with the pre-trained decoder turning predicted codes back into signals. The quantization layer is what converts a regression problem into classification and what forces the model to reuse a small set of prototypical patterns, which the paper argues is the key to capturing sparse activations.","core_discovery":"The central discovery is that vector quantization can convert neural response forecasting into a code-prediction task that handles sparse activations better than regression. QuantFormer first learns, through masked auto-encoding, to map patches of calcium traces to one of 32 learnable codebook embeddings; at forecasting time the model masks the response window, classifies each masked patch into a code, and feeds the predicted codes through the frozen decoder to produce a continuous forecast. With this scheme, the paper reports slightly better stimulus-response classification than competitors (about 77.4% accuracy on the selected Allen containers) and better forecasting metrics under its gradient-sum normalization, with generalization across stimulus categories and subjects.","pith_inferences":["Because the forecast is decoded from discrete codes by a frozen decoder, one could swap the decoder for one trained on deconvolved or spiking signals and keep the encoder's learned dynamics, a route the paper leaves implicit.","The neuron-token design suggests continual learning: previously unseen neurons could be added with fresh tokens and prompt-only training, which would be a direct test of whether the encoder stores general V1 dynamics.","The 32-code vocabulary invites a stronger causal test: if codes are real neural primitives, perturbing a single predicted code should change the decoded trace in a stereotyped way, which is checkable in the paper's own reconstruction framework.","The same architecture could be pointed at other sensory cortices or species by retraining tokens, since nothing in the quantization or masking machinery is mouse-specific."],"forward_implications":["If the claim is right, one model can forecast activity for any number of neurons by adding a new [NEURON] token, avoiding per-neuron or fixed-population models.","Forecasting as code classification should generalize to other sparse neural signals, since the quantization acts as a built-in sparsity prior; the paper shows pre-training transfers across stimulus types and mice.","Closed-loop optogenetics becomes more feasible: predicted future activity in the response window can be produced from the baseline trace alone, the input available in real time.","The 32-code vocabulary gives a compact, interpretable intermediate representation, and the paper shows neuron embeddings cluster by response statistics such as activation frequency."],"supporting_citations":[{"why":"Supplies the Allen Visual Coding dataset with raw fluorescence traces and stimulus structure used for pre-training and evaluation.","marker":"[41]"},{"why":"Provides the deep-learning cell-identification and trace-extraction pipeline and the 11-container selection used in the experiments.","marker":"[43]"},{"why":"Defines the 10% gain rule that labels a neuron as active, used for the classification downstream task and data balancing.","marker":"[21]"},{"why":"Provides the vector-quantization training machinery, including the straight-through estimator and codebook/commitment losses.","marker":"[61]"},{"why":"Supplies the masked auto-encoding pre-training scheme that QuantFormer adapts to time-series patches.","marker":"[40]"},{"why":"Provides the precedent that posing a continuous generation problem as classification eases optimization, the rationale for quantization.","marker":"[38]"},{"why":"Informer is a time-series transformer baseline that QuantFormer must beat in forecasting and classification.","marker":"[47]"},{"why":"Crossformer is the strongest multivariate transformer baseline; its channel-attention design is contrasted with QuantFormer's neuron tokens.","marker":"[49]"},{"why":"Autoformer is a decomposition transformer baseline included in the forecasting comparisons.","marker":"[65]"},{"why":"BrainLM is the neural foundation-model baseline pre-trained on fMRI, tested both frozen and fine-tuned.","marker":"[55]"}],"fun_headline_variants":["Forecasting neural activity via code classification","Quantized transformer predicts brain activity","Mouse cortex forecasts as code prediction","Learned quantization for neural forecasting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gradient-sum normalization used for forecasting metrics is the correct evaluation protocol: the paper's Appendix Table A-4 shows that with standard un-normalized MSE, MAE, SMAPE, Pearson correlation and SSIM, a constant-mean baseline beats QuantFormer on every metric, and the main Table II omits that baseline.","fun_headline_variants_meta":{"raw":{"variants":["Forecasting neural activity via code classification","Quantized transformer predicts brain activity","Mouse cortex forecasts as code prediction","Learned quantization for neural forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1410,"prompt_tokens":923,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":539,"tokens_out":487,"duration_ms":6195,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:36.392785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the forecasting comparison on the same Allen containers with un-normalized responses and include a constant-mean baseline; Table A-4 already provides the result, showing the baseline winning on MSE, MAE, SMAPE, Pearson correlation and SSIM, so the forecasting superiority claim would fail under standard evaluation unless the normalization is defended as the right protocol.","supporting_citations":[{"cited_title":"A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex,","cited_arxiv_id":null,"evidence_quote":"Supplies the Allen Visual Coding dataset with raw fluorescence traces and stimulus structure used for pre-training and evaluation."},{"cited_title":"A deep-learning approach for online cell identification and trace extraction in functional two-photon calcium imaging,","cited_arxiv_id":null,"evidence_quote":"Provides the deep-learning cell-identification and trace-extraction pipeline and the 11-container selection used in the experiments."},{"cited_title":"Ultrasensitive fluorescent proteins for imaging neuronal activity,","cited_arxiv_id":null,"evidence_quote":"Defines the 10% gain rule that labels a neuron as active, used for the classification downstream task and data balancing."},{"cited_title":"Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks,","cited_arxiv_id":null,"evidence_quote":"Provides the vector-quantization training machinery, including the straight-through estimator and codebook/commitment losses."},{"cited_title":"Masked autoencoders are scalable vision learners,","cited_arxiv_id":null,"evidence_quote":"Supplies the masked auto-encoding pre-training scheme that QuantFormer adapts to time-series patches."},{"cited_title":"Informer: Beyond efficient transformer for long sequence time-series forecasting,","cited_arxiv_id":null,"evidence_quote":"Informer is a time-series transformer baseline that QuantFormer must beat in forecasting and classification."},{"cited_title":"Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,","cited_arxiv_id":null,"evidence_quote":"Crossformer is the strongest multivariate transformer baseline; its channel-attention design is contrasted with QuantFormer's neuron tokens."},{"cited_title":"Brainlm: A foundation model for brain activity recordings,","cited_arxiv_id":null,"evidence_quote":"BrainLM is the neural foundation-model baseline pre-trained on fMRI, tested both frozen and fine-tuned."}],"review_version":1}