{"id":"5e682250-09bb-4539-8cec-d14f401c018f","arxiv_id":"2507.03899","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A Transformer model predicts the next-visit Alzheimer's diagnosis from longitudinal visit sequences and outperforms RNN baselines on conversion cases, especially when few visits are available.","lead":"The authors train a Transformer to predict a patient's Alzheimer's disease stage at the next clinic visit from a sequence of past visits. It beats recurrent neural networks at catching patients who convert to a more severe stage, at the cost of worse predictions on patients whose stage stays stable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Imputation protocol is underspecified; if the minimalRNN filler saw test labels, all converter gains could be inflated by label leakage.","rationale":"The reader's weakest assumption is precisely the model-filling imputation protocol and its potential to leak target information. This is the most load-bearing concern because it directly threatens the paper's central comparative claim: if imputation saw test labels, the observed converter advantage of Transformers over RNNs could be an artifact of differential exploitation of leaked features. Other issues (e.g., overlapping folds in statistical tests, exclusion of reverters) are real but less likely to reverse the large, consistent converter gains; the leakage concern, if confirmed, would invalidate the headline result. The paper does provide code, so the proposed check is feasible and should be required before accepting the claim. Because the reader already conditioned acceptance on clarifying the imputation protocol, my assessment does not move the verdict; it remains CONDITIONAL pending this verification.","tokens_in":12791,"tokens_out":3093,"duration_ms":31524,"concrete_test":"Inspect the supplemental code to determine whether the minimalRNN imputer is fit on the entire dataset before the 10-fold split or separately on each training fold. Then re-run the full pipeline with the imputer trained only on each training fold's sequences (including their diagnoses) and applied to the corresponding test fold. Compare converter BCA and the Transformer-vs-best-RNN gap; if the gap shrinks materially or loses significance, the reported converter gains are partly an artifact of leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4) is that Transformer-based models significantly outperform RNNs in identifying converter sequences. This comparison is only valid if the features used at test time do not encode information about the target diagnosis. Section 2.2 describes a 'model filling' imputation method that fills missing values using predictions from a minimalRNN, but it never states whether this imputer was trained on training folds only or on the full dataset including test sequences and their diagnoses. With per-feature missingness ranging from 29% to 83%, most input values are synthetic; if the imputer was fit on the full dataset, the filled test features can leak the very label the model is asked to predict. Because the Transformer attends to all positions simultaneously, it may exploit such leakage more effectively than the RNN baselines, directly inflating the reported converter-detection gap. Even without explicit label leakage, fitting the imputer on the full dataset before the 10-fold split is a form of test-data leakage that biases the evaluation. The text does not rule this out, and the provided code would settle it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a Transformer-based model for predicting the next-visit clinical diagnosis (CN, MCI, AD) from a subject's longitudinal visit history in the TADPOLE/ADNI data. The authors construct stable and converter sequences, handle missing values by a 'model filling' imputation step using a minimalRNN, and compare the Transformer to LSTM, GRU, minimalRNN, and a Time Series Transformer under 10-fold cross-validation. They report that Transformer-based models significantly outperform RNNs on converter sequences, particularly for short histories, at the cost of somewhat lower stable-sequence accuracy, and they include ablations on visit-history length and feature categories.","tokens_in":13020,"tokens_out":11272,"duration_ms":125622,"significance":"If the reported effects are valid, the paper provides a practically useful comparison showing that attention-based models can identify AD progression converters better than RNN baselines, with public data, clearly defined stable-versus-converter sequence construction, and code availability. The per-group evaluation and the feature/ablation analyses are also strengths. However, the validity of the central claim depends on preprocessing details that are currently underspecified, especially the imputation protocol and the temporal auxiliary feature, so the contribution is conditional on those being resolved.","major_comments":[{"comment":"The imputation protocol is underspecified in a way that can directly invalidate the central claim. Table 1 shows per-feature missingness between 29.40% and 83.38%, so most values in a typical input vector are filled. The text only says that 'model filling' uses predictions from a minimalRNN model; it does not state whether this imputer was trained on each training fold, on the full dataset, or with access to the target diagnosis. If the minimalRNN was fit on the full dataset before the 10-fold construction, the filled test values can encode the label at the target visit, and because Diagnosis is itself one of the 23 features, leakage would preferentially inflate converter detection and the reported advantage over RNNs. Please specify the exact protocol, verify in code that no test fold contributes to imputer training, and consider an ablation without model filling.","section":"§2.2, Table 1"},{"comment":"The auxiliary feature 'number of months until the final visit' may constitute target leakage. Each generated sequence ends at the target visit (the conversion visit for converters and the last stable visit for stable sequences), so for input visits the feature equals the time remaining until the visit whose diagnosis is being predicted. In a prospective deployment, the date of the next visit is sometimes known, but the feature also encodes follow-up duration, which may differ systematically between converters and stable subjects; the manuscript does not show that this feature is available at inference time or that results are robust to its removal. Please add an ablation without this feature and discuss its clinical availability.","section":"§2.2, Definitions 1-3, §2.4"},{"comment":"The description of the cross-validation procedure is internally inconsistent. The text first says each fold is divided so that 10% of each group is in the test split and 90% is in the train split, then says 'We split the data temporally, ensuring that each fold has an equal proportion of sequences from each group.' A temporal split by visit date or sequence length will not, in general, preserve equal group proportions, and a stratified split is not purely temporal. Please specify the exact fold-generation algorithm, including how ties in dates and multiple sequences per subject are handled, and state whether the imputer is fit inside each fold. Without this, the standard errors and p-values across folds are not well defined.","section":"§2.5"},{"comment":"The statistical tests are not appropriate for the data structure. All significance claims use a two-sample Welch's t-test with n1=n2=10 on fold-level metrics. Fold metrics are paired across models because the same fold's test sequences are scored by every model, and they are not independent across folds because the training sets for different folds overlap heavily. A paired test or a mixed-effects model with fold and model as factors is the appropriate analysis; moreover, many metrics and subgroup comparisons are tested without multiple-comparison control. The very small p-values for converter accuracy would likely survive a more valid test, but the full set of claimed differences in Tables 3-6 should be re-evaluated.","section":"§3, first paragraph"},{"comment":"The comparison does not control for model capacity. The Transformer is reported to have 6,603,267 parameters, but no parameter counts or compute budgets are given for LSTM, GRU, minimalRNN, or TST. If the RNN baselines have far fewer parameters, the converter advantage could be due to capacity rather than the attention mechanism. Please report parameter counts for every model and, ideally, include a capacity-matched comparison (e.g., larger RNNs or a smaller Transformer).","section":"§2.4, Table 2"}],"minor_comments":[{"comment":"Definition 1 includes CN-to-AD conversions, but the text later says converter sequences are generated only from CN-to-MCI and MCI-to-AD conversions; please clarify whether CN-to-AD sequences exist in the dataset and how they are labeled.","section":"§2.3"},{"comment":"The balanced dataset is created by randomly discarding stable sequences; please report the random seed or run multiple resamplings to ensure results are not artifacts of one draw.","section":"§2.3"},{"comment":"The description of the Transformer architecture is incomplete; state whether the model is encoder-only or encoder-decoder, how visits are embedded, and what positional encoding is used.","section":"§2.4"},{"comment":"F1 scores are reported only for overall performance; reporting F1 for stable and converter subsets would aid interpretation of the trade-off.","section":"Tables 2 and 5"},{"comment":"The figures are hard to read in the provided version; please enlarge axis labels and define group numbers in the captions.","section":"Figures 2, 5-7"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is potentially solid, but the imputation protocol and the temporal feature are load-bearing and currently underspecified. Because the code is available, the authors can resolve the leakage question by verification and by adding an ablation; I therefore recommend a major revision rather than rejection. The statistical testing issue and the capacity matching issue should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's actual contribution is a focused empirical comparison: on TADPOLE, a vanilla Transformer and the TST beat LSTM, GRU, and minimalRNN at detecting converter sequences, while doing worse on stable sequences, and this advantage is concentrated in short visit histories. That is a new finding, not a new architecture, and the comparison is set up cleanly with raw and balanced datasets, multiple metrics, and ablations. The stable-versus-converter split is the right lens for this clinical question, and the results are internally consistent across the two datasets. They also include code, which is more than many papers in this space do.\n\nThe soft spot is exactly where the stress test lands: Section 2.2 says missing values are filled by 'model filling' with a minimalRNN, but never states whether that model was trained on training folds only or on the full dataset including test sequences and their diagnoses. With per-feature missingness ranging from 30% to 83%, most input values are synthetic, and if the imputer saw the test labels, every reported gain could be inflated—especially for the Transformer, which attends to all positions at once. The text does not rule this out. This is the load-bearing concern, and it is a genuine one. The provided code would settle it, but the paper as written is incomplete on this point.\n\nThe other issues are minor by comparison. The 'temporal split' description is vague and seems to conflict with the stratified folds. The Welch t-tests treat the ten folds as independent samples, which is common but not strictly correct; a confidence interval that accounts for fold dependence would be more convincing. Dropping reverters and post-conversion visits makes the converter definition cleaner but limits generalizability—worth stating as a limitation, which the paper mostly does.\n\nThis paper deserves a serious referee. The central question is empirical: does the Transformer advantage survive a properly specified imputation protocol? If yes, this is a useful incremental contribution for the clinical ML community. If no, it is a cautionary tale about leakage. The authors should be asked to describe the imputation fit, report whether it saw test data, and re-run with subject-disjoint splits if needed. I would flag it for major revision, not desk reject.","headline":"A useful, internally consistent Transformer-vs-RNN comparison for AD conversion prediction, but the underspecified model-filling imputation protocol leaves the headline result conditional on a fix.","tokens_in":13500,"tokens_out":1975,"would_cite":true,"duration_ms":21904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformer models that read a patient's full visit history at once flag Alzheimer's stage transitions significantly better than recurrent networks, at a small cost in stable-patient accuracy.","keywords":["Alzheimer's disease progression","Transformer","longitudinal prediction","converter sequences","MCI to AD conversion","recurrent neural networks","model filling imputation","TADPOLE dataset"],"falsifier":"Re-run the 10-fold cross-validation with the model-filling imputer trained separately on each training fold and applied to the held-out test fold, then compare the Transformer's converter accuracy against the RNN baselines; if the advantage shrinks to below statistical significance, the reported converter gains were artifacts of test-set leakage through imputed values.","tokens_in":12507,"feed_emoji":"🧠","tokens_out":7179,"duration_ms":74078,"temperature":0.7,"pith_summary":"This paper proposes a Transformer model that takes a patient's sequence of clinical visits as input and predicts the diagnosis at the next visit, comparing it against LSTM, GRU, minimalRNN, and a time-series Transformer on the TADPOLE dataset. The central claim is that attention-based models are significantly better than RNNs at identifying converter sequences—the visits where a patient moves from CN to MCI or from MCI to AD—while being slightly worse on stable sequences. The authors argue that this trade-off is the right one for Alzheimer's care, because missing a conversion is clinically costlier than a false alarm. The paper also finds that cognitive test scores carry most of the predictive signal, and that the converter advantage is strongest for short visit histories, which is exactly the setting where early warning is needed.","feed_headline":"Transformer beats RNNs at catching Alzheimer's stage changes","feed_subtitle":"Attention models spot CN-to-MCI and MCI-to-AD transitions RNNs miss, especially from short visit histories.","key_machinery":"The load-bearing mechanism is multi-head self-attention over visit tokens: each visit in the sequence is embedded into a 256-dimensional vector and attends to all other visits, so the model can compare the current trajectory against earlier states without compressing history into a single hidden state. This is supported by three auxiliary pieces: 'model filling' imputation, which fills missing feature values using predictions from a minimalRNN; a temporal feature representing months until the final visit; and the construction of converter versus stable sequences, where a converter sequence ends at a diagnosis change and a stable sequence ends at the same diagnosis as the previous visit. The evaluation centers on BCA, the mean of per-class sensitivity and specificity averaged over CN, MCI, and AD, computed separately on stable and converter subsets.","core_discovery":"On the paper's own terms, the discovery is that Transformer-based architectures allocate their predictive effort differently from RNNs: the RNNs in the study achieve high accuracy mostly by predicting that stable patients remain stable, with converter accuracy on the raw dataset as low as 0.129 for minimalRNN and 0.217 for LSTM, while the Transformer reaches 0.648 and the time-series Transformer 0.694. On a class-balanced dataset the Transformer's converter accuracy rises to 0.792 versus 0.439 for the best RNN, and the differences in accuracy, BCA, sensitivity, and specificity are statistically significant. The advantage is concentrated in the earliest visit groups and fades after roughly four prior visits. Ablations show that removing cognitive scores cuts BCA by about 16 percent relative, while removing volumetric MRI or biomarkers changes little, and training on only the last one or four visits degrades BCA by 6.6 and 5.0 percent respectively. The authors interpret the overall pattern as evidence that attention-based models are better suited for early detection, even though they underperform RNNs on stable patients.","pith_inferences":["A critical open question the paper leaves unresolved is whether the minimalRNN imputer was fit on training folds only; if it saw the full dataset, the filled test-visit features could encode the target diagnosis and inflate all reported converter gains.","The study excludes 'reverter' subjects and any visits after a second conversion, so the model's real-world utility for patients with fluctuating or improving diagnoses is untested.","If the imputation-leakage concern is resolved and the effect persists, an immediate next experiment is to test the same architecture on external cohorts with different visit schedules and feature missingness patterns to see whether the short-history advantage generalizes.","The feature-ablation result implies that expensive and invasive biomarker measurements contribute little beyond cognitive scores in this model, which, if confirmed, could reduce the burden and cost of longitudinal Alzheimer's monitoring."],"forward_implications":["If attention-based models are genuinely better at converter detection, early-warning systems for Alzheimer's disease should prefer them over RNN baselines, since catching a CN-to-MCI or MCI-to-AD transition is the clinically important event.","The converter advantage appears for both the raw and class-balanced datasets, so it is not merely a side effect of majority-class bias.","The largest gains come from short visit histories (groups 1-3), meaning a model trained on two to four prior visits could flag high-risk patients before long-term longitudinal data accumulate.","Because cognitive scores dominate the predictions, data-collection protocols could in principle be streamlined around cognitive testing without sacrificing most of the model's accuracy.","Longer visit histories do not automatically yield better predictions, so future model design needs to address data sparsity and irregular spacing in later visit groups rather than assuming more data always helps."],"supporting_citations":[{"why":"Supplies the TADPOLE longitudinal dataset and the recommended feature list used in all experiments.","marker":"Marinescu et al., 2018"},{"why":"Provides the 'model filling' imputation method and the recurrent training strategy the baselines follow.","marker":"Nguyen et al. (2019)"},{"why":"Defines the minimalRNN architecture used both as the imputer and as one of the baseline models.","marker":"Chen (2018)"},{"why":"Introduces the Transformer architecture on which the proposed model is built.","marker":"Vaswani et al. (2017)"},{"why":"Defines the Time Series Transformer used as the newer Transformer-based comparison baseline.","marker":"Zerveas et al. (2021)"},{"why":"Defines the LSTM baseline.","marker":"Hochreiter and Schmidhuber (1997)"},{"why":"Defines the GRU baseline.","marker":"Cho et al. (2014)"},{"why":"Defines the multiclass AUC metric used for evaluation.","marker":"Hand and Till (2001)"}],"fun_headline_variants":["Transformer beats RNNs on Alzheimer's stage-change detection","Attention model spots early Alzheimer's stage shifts RNNs miss","Transformer improves early Alzheimer's progression prediction","Transformer outperforms RNNs at predicting AD converters","Short visit history? Transformer still predicts AD stage change"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the missing-value imputation step fills in realistic measurements for features missing in 29 to 83 percent of visits without leaking the diagnosis being predicted, and it does not state whether that imputer was trained only on the training folds.","fun_headline_variants_meta":{"raw":{"variants":["Transformer beats RNNs on Alzheimer's stage-change detection","Attention model spots early Alzheimer's stage shifts RNNs miss","Transformer improves early Alzheimer's progression prediction","Transformer outperforms RNNs at predicting AD converters","Short visit history? Transformer still predicts AD stage change"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1367,"prompt_tokens":968,"completion_tokens":399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":584,"tokens_out":399,"duration_ms":5119,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:59:54.784081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 10-fold cross-validation with the model-filling imputer trained separately on each training fold and applied to the held-out test fold, then compare the Transformer's converter accuracy against the RNN baselines; if the advantage shrinks to below statistical significance, the reported converter gains were artifacts of test-set leakage through imputed values.","supporting_citations":[{"cited_title":"Alexander, Jiashi Feng, and B","cited_arxiv_id":null,"evidence_quote":"Provides the 'model filling' imputation method and the recurrent training strategy the baselines follow."},{"cited_title":"Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N","cited_arxiv_id":null,"evidence_quote":"Introduces the Transformer architecture on which the proposed model is built."},{"cited_title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","cited_arxiv_id":null,"evidence_quote":"Defines the GRU baseline."}],"review_version":1}