{"id":"911fc855-11cb-4d35-ae19-2e6506ce4e19","arxiv_id":"2507.14847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"TALE-EHR uses a learned polynomial time weight inside transformer attention, plus pre-trained language model embeddings of medical codes, and reports higher predictive accuracy than six EHR baselines on MIMIC-IV and PIC.","lead":"This paper presents TALE-EHR, a transformer model that adds a learnable time-weighting signal to attention so past medical events influence predictions according to how long ago they occurred. It reports better disease and next-code predictions than several baseline models on the MIMIC-IV and PIC hospital datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central SOTA claim rests on an unspecified disease-onset evaluation protocol: negatives and their index times are undefined, so reported AUROC gaps may reflect temporal leakage rather than model quality.","rationale":"The reader's weakest assumption identifies the same load-bearing point, and the full text supports it. The architecture is described clearly, and the ablation gives some internal evidence for the time-aware component, but the central quantitative claim is empirical and depends entirely on a fair disease-onset evaluation. The absence of any negative-patient definition, index time, or censoring rule is the weakest link because a small change in that protocol can change AUROC rankings. A matched-index negative control is a concrete, feasible check that would settle whether the reported gains are due to temporal modeling or to label leakage. The duplicate rows and missing code/data prevent independent verification but do not, by themselves, prove the results wrong. The existing CONDITIONAL verdict therefore remains appropriate: acceptance should hinge on clarifying the evaluation protocol and ideally releasing code that specifies negative sampling.","tokens_in":906,"tokens_out":746,"duration_ms":42733,"concrete_test":"Run the MIMIC-IV nine-disease experiment under two negative-control protocols: (A) negatives are patients without the disease in the full record, with histories taken from the start as implied by the preprint; (B) negatives are patients without the disease diagnosis up to a matched index time (e.g., matched on number of preceding events or follow-up duration), with disease codes removed from positive pre-onset inputs and histories censored at the index. Recompute Table 1 AUROC under protocol (B). If the average drops materially (e.g., by more than 0.02 AUC) or TALE-EHR no longer beats Cehr_Bert on the same diseases, the headline SOTA claim is unsupported. As a secondary check, release the exact negative definitions and index-time sampling code, since without this information no reader can reconstruct the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that TALE-EHR reaches an average AUC of 0.926 across nine diseases and Acc@5 of 0.902 on next-code prediction. For the disease AUC part, the load-bearing condition is that binary labels are constructed without temporal leakage. Section 3.4 states only that the model processes 1024 clinical events before the target event, 'e.g., the first occurrence of a disease,' and Algorithm 1 line 19 says 'Extract histories before disease onset time.' The paper never defines negative patients, the index time for patients who never develop the disease, or how censoring is handled. Appendix F's UMAP description says red points are patients who 'eventually developed' the disease and blue points are patients who 'did not,' which suggests negative labels are derived from the full record. If negative histories are drawn from the whole record while positive histories stop at first disease onset, the model can separate classes by record composition and length, and negatives can contain events occurring after their notional prediction time. This would inflate AUROC independently of the proposed time-aware attention. The same protocol is used on PIC (Table 3), so the generalization claim inherits the same risk. Tables 1 and 4 also contain duplicate or copy-paste rows, preventing independent verification of baseline strengths. This is a reporting gap and a correctness risk, not a demonstrated error, but until negatives and index times are specified and a censored-at-index protocol is used, the state-of-the-art claim is not securable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes TALE-EHR, a Transformer-based marked point-process model for EHR sequences. Its core idea is to multiply the scaled dot-product attention score by a learned sigmoid-polynomial w(t) of the continuous inter-event time, and to embed medical codes with fixed BGE embeddings of their textual descriptions. The model is pre-trained with a joint least-squares intensity loss and a focal code-prediction loss, then fine-tuned for disease-onset and clinical benchmark tasks. On MIMIC-IV the paper reports an average disease-prediction AUC of 0.926 across nine conditions, gains on readmission and length-of-stay benchmarks, and Acc@5 of 0.902 on next-code prediction; on PIC it reports consistent gains across four pediatric tasks. Ablations attribute most of the improvement to the time-aware mechanism, with a smaller contribution from the LLM-derived embeddings.","tokens_in":15924,"tokens_out":9513,"duration_ms":106345,"significance":"The proposed mechanism is a reasonable and testable modification of Transformer attention: instead of adding time embeddings or using discrete time tokens, it directly modulates attention logits by a learned continuous function of the time gap. The architecture and losses are described in enough detail that the method could in principle be reimplemented, and I find no derivation loop: w(t) is learned on training labels and evaluated on held-out data, and the BGE embeddings are a fixed external encoder. The ablation design is sensible and the high-level claims are plausible. However, the empirical evidence as printed is not yet verifiable: the disease-onset label protocol is underspecified, Section 4.3 points to a nonexistent Appendix E, Tables 1 and 4 contain duplicate or malformed rows, no code or data are released, and the Acc@K numbers in Table 7 are not consistent with the standard cumulative definition. The contribution is worth considering, but the reported SOTA claims need stronger reporting and, where possible, re-analysis under a clearly defined evaluation protocol.","major_comments":[{"comment":"The disease-onset evaluation is the load-bearing support for the paper's headline result (average AUC 0.926, Table 1), but the label construction is not specified. The text says only that the model uses 1024 clinical events before the target event, 'e.g., the first occurrence of a disease,' and Algorithm 1 says 'Extract histories before disease onset time.' It never defines negative patients, an index time for patients who never develop the disease, or how censoring is handled. Appendix F's description of negatives as patients who 'did not' develop the disease suggests labels are assigned from the full record. If positive histories are truncated at first onset while negative histories are drawn from the entire record, then sequence length/composition and post-index events become available to the classifier, which can inflate AUROC independently of the time-aware attention. Please specify the negative-label definition and a single index time for all patients, add a censoring-aware evaluation (e.g., fixed prediction horizon or time-matched negative sampling), and state whether the 1024-event window is padded or truncated; the same protocol applies to the PIC results in Table 3.","section":"Section 3.4; Algorithm 1 (line 19)"},{"comment":"The empirical tables contain errors that prevent verification: in Table 1 the HiTANet row for Hyperlipidemia is identical (up to reporting precision) to its Arteriosclerosis row, and in Table 4 the Hyperlipidemia row appears twice, with malformed entries such as '0.942±.0060.937±.006' and '0.964±.0050.957±.004'. Combined with the absence of released code or data, this means the reported baseline strengths and ablation comparisons cannot be checked from the manuscript. Please correct the tables, verify that every row corresponds to a distinct disease and run, and release code/data or detailed evaluation scripts so that the comparisons are reproducible.","section":"Tables 1 and 4"},{"comment":"Section 4.3 refers to Appendix E for the definitions of Acc@K, macro F1, and Recall, but no Appendix E exists in the manuscript. This is not merely a missing cross-reference: in Table 7 every model's Acc@K decreases as K grows (e.g., TALE-EHR: 0.902, 0.896, 0.862), which is incompatible with the standard cumulative top-K recall definition. Please supply Appendix E or define the metric precisely, and state whether Acc@K is cumulative recall, precision at K, or an exact-set match; the same clarification is needed for the multi-label targets and the meaning of 'next visit.'","section":"Section 4.3; Table 7"},{"comment":"The polynomial order (Poly=5) is selected after inspecting the ablation results reported in Table 4. If these AUROC values are computed on the test set, then the choice of w(t)'s degree is made using test information, and the reported advantage of Poly=5 over the alternatives may be optimistic. Please state explicitly whether the ablation comparisons were made on a validation set, and if not, provide validation-based model selection or report the selection procedure.","section":"Section 6; Table 4"}],"minor_comments":[{"comment":"The symbol γ is used both for the focal-loss exponent (γ=2.0) and for the weight of L_code in the combined loss L = L_time + γ L_code; please use different symbols for these two quantities.","section":"Section 3.3"},{"comment":"BGE-M3 is described as a 'pre-trained LLM,' but it is a text embedding model rather than an autoregressive or generative language model; please adjust the terminology to 'pre-trained text encoder' or similar.","section":"Section 3.1"},{"comment":"The pseudocode does not specify how patient histories with fewer than 1024 events are handled, nor how the 'current time t' is chosen for each training example; please make these details explicit.","section":"Algorithm 1"},{"comment":"The discussion of disease-specific decay rates (acute vs. chronic) is qualitative and visual; a quantitative summary, such as fitted decay constants or a comparison of w(t) across diseases, would strengthen the claim.","section":"Appendix D"},{"comment":"The reference list contains duplicate and inconsistent entries (e.g., two HiTANet entries with different venues, and multiple Retain entries); please consolidate and correct the bibliography.","section":"References"},{"comment":"The median frequency for CCS codes is reported as 1.587, while surrounding entries use comma separators; this is likely a formatting error and should be corrected.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The main uncertainty is the disease-onset label protocol. If negatives are constructed from the full record while positives are truncated at first onset, the reported AUROC gains could be an artifact. I would request the exact label-construction procedure and a re-analysis with a fixed prediction horizon or time-matched negative sampling before accepting the empirical claims. The duplicate/malformed rows and the missing Appendix E are easy to fix but suggest that the empirical sections were not carefully proofread; corrected tables and either code or detailed evaluation scripts should be required."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2507.14847. The contribution is genuinely incremental: a learnable sigmoid-polynomial time weight w(t) multiplied into self-attention scores, plus frozen BGE embeddings of code descriptions. That combination is a legitimate extension of HiTANet and Cehr_BERT rather than a copy, and the architecture is clearly described. The two-dataset evaluation (MIMIC-IV and PIC) is substantial, and the ablation honestly shows the time-aware mechanism matters more than the LLM embeddings. That is real work and the paper does it reasonably well.\n\nThe soft spots are real. Most important: the disease-onset evaluation protocol is underspecified. Section 3.4 says the model takes 1024 events before the target event, and Algorithm 1 says 'extract histories before disease onset time,' but there is no definition of negative patients, the index time for patients who never develop the disease, or how censoring is handled. Appendix F's UMAP description says red points 'eventually developed' and blue 'did not,' which suggests negatives are labeled from the full record. If negative histories are drawn from the full record while positive histories stop at first onset, class separation can come from record composition and length rather than model quality. That ambiguity is load-bearing for the headline AUC of 0.926, and the PIC evaluation inherits it.\n\nSecond, the tables need a cleanup. Table 1 has identical HiTANet AUPRC/F1 for Arteriosclerosis and Hyperlipidemia, and Table 4 repeats Hyperlipidemia instead of listing Hypertension. Almost certainly copy-paste errors, but they block independent verification. Third, no code or data release, so the state-of-the-art claim cannot be checked.\n\nNone of this is a demonstrated error, and the architecture is plausible. But the SOTA claim as printed is not securable. A serious venue should ask for the negative sampling and index time to be specified, an evaluation censored at index, corrected tables, and a code release.\n\nWho benefits: people working on time-aware transformers for EHR will treat this as related work and might build on the w(t) idea. It deserves a serious referee, with the expectation of major revision.","headline":"An incremental but plausible EHR transformer whose headline AUC numbers cannot be verified until the disease-onset negative sampling is specified and the tables are fixed.","tokens_in":16442,"tokens_out":2770,"would_cite":true,"duration_ms":28735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62M30"],"pacs":[],"model":"deepseek-v4-flash","headline":"TALE-EHR folds learnable continuous time gaps into Transformer attention and reports state-of-the-art results for EHR disease and code prediction on MIMIC-IV and PIC.","keywords":["electronic health records","time-aware attention","temporal point process","Transformer","medical code prediction","disease onset prediction","LLM code embeddings","MIMIC-IV"],"falsifier":"Re-run the nine disease tasks on MIMIC-IV with explicit negative-patient construction: each control is assigned the same index time as a matched case (or a random index from the control's own record) and the prediction window ends at that index; drop all cases with fewer than 1024 prior events exactly as the paper does. If TALE-EHR's AUROC advantage over the strongest baseline, Cehr_Bert, shrinks to within noise under this leakage-free labeling, then the reported superiority is an artifact of label construction rather than of time-aware attention.","tokens_in":15418,"feed_emoji":"🏥","tokens_out":7625,"duration_ms":78614,"temperature":0.7,"pith_summary":"The paper claims that the main obstacle to accurate EHR forecasting is not the sequence of clinical codes but the irregular, continuous time between events, and that a Transformer can be made to respect that time by folding a learnable function of the time gap directly into each attention score. It introduces TALE-EHR, which pairs this time-aware attention with embeddings that a general-purpose text encoder produces from standard code descriptions, so the model starts with clinical semantics rather than random code vectors. On MIMIC-IV the authors report an average area under the ROC curve of 0.926 across nine diseases, top Acc@5 of 0.902 for next-code prediction, and gains on readmission and length-of-stay benchmarks; on the PIC pediatric database they report the best AUROC, AUPRC, and F1 across pneumonia, heart malformations, mortality, and ICU stay longer than seven days. If these results hold, the practical message is that explicit continuous time itself, not just richer code representations, carries much of the predictive signal in EHRs.","feed_headline":"Time-aware attention lifts EHR predictions past previous best","feed_subtitle":"Feeding real time gaps between events into attention beats baselines on MIMIC-IV and PIC disease and code tasks.","key_machinery":"The key object is the time-aware attention function $A(Q,K,t)=\\mathrm{softmax}(Q^\\top K/\\sqrt{d}\\, w(t))$ with a learnable polynomial-gated weighting $w(t)=\\sigma(a_0+a_1 t+\\dots+a_s t^s)$ evaluated on the continuous gap $|t_j-t_k|$ between events. A sigmoid keeps the temporal weight in $[0,1]$, so the mechanism can downweight distant events without erasing them; the polynomial basis gives a flexible, interpretable decay curve that is fitted jointly with the rest of the network. A hierarchical attention step then forms the patient history representation $h_t=\\sum_j \\alpha_j(t)E_{c_j}(t)$, where the same temporal weighting modulates how much each event contributes to the current state. This machinery lets the model treat time as part of the attention computation itself rather than as a preprocessing step, and it is trained jointly with a temporal point process loss for event timing and a focal loss for code prediction.","core_discovery":"TALE-EHR's central claim is that explicitly modeling the continuous time difference between clinical events inside the attention mechanism is the decisive ingredient for EHR sequence modeling. Rather than relying on sequence order, separate time embeddings, or discretized time tokens, the model multiplies each softmax attention weight by a learnable temporal function $w(t)=\\sigma(a_0+a_1 t+\\dots+a_5 t^5)$ of the time gap, so that a past event's influence decays or persists depending on the condition being predicted. The learned weights are disease-specific: the paper shows rapid decay for acute kidney injury and gradual decay for chronic conditions such as type 2 diabetes and heart failure. The same time-aware weighting is used twice, first inside code-level attention and again to aggregate an entire patient history into a single vector used for downstream prediction. On top of this, the model freezes the embeddings that a pre-trained general-purpose text encoder produces from the textual descriptions of 12,232 medical codes, and the paper's ablation attributes a smaller but consistent accuracy gain to these embeddings, with the time-aware mechanism contributing the larger share.","pith_inferences":["One testable extension not reported in the paper: the learned $w(t)$ curves could be read as per-disease forgetting functions, and plotting them across hospitals or across code systems would show whether the model is learning clinical time constants or dataset artifacts.","A practical diagnostic follows from the paper's own design: if $w(t)$ saturates to a constant near 1 for all diseases after training, the time-aware mechanism would collapse to standard self-attention, so routinely plotting the learned weight curve would reveal when the claimed mechanism is actually inert.","Because the embedding component contributes less than the temporal component, a cheaper variant that replaces the general-purpose text encoder with hand-curated code vectors might retain most of the gain; this is directly checkable and would matter for deployment in low-resource settings.","The unclear negative-patient definition implies that the same framework should be re-benchmarked under a public, standardized outcome-labeling protocol before the absolute AUC numbers are used to set clinical decision thresholds."],"forward_implications":["Disease forecasting: TALE-EHR reports the highest AUROC on all nine MIMIC-IV diseases, including 0.961 for acute kidney injury, so disease-onset screening could move to a model that uses continuous event timing.","Next-code prediction: the reported Acc@5 of 0.902 and F1 of 0.850 across 12,232 codes suggest better auto-completion of the medical record, reducing missed future codes.","Benchmark tasks: reported 30-day readmission AUROC of 0.762 and next-visit length-of-stay over 7 days AUROC of 0.759 beat all baselines, indicating the representation transfers to outcomes beyond code prediction.","Pediatric generalization: on PIC, the best AUROC on pneumonia (0.945), heart malformations (0.906), mortality (0.934), and ICU stay over 7 days (0.897) suggests the method transfers across patient populations.","Component attribution: the ablation shows that removing time-awareness costs more accuracy than replacing LLM embeddings with random vectors, so continuous temporal modeling is the primary driver of the reported gains."],"supporting_citations":[{"why":"Supplies the pre-trained general-purpose text encoder whose frozen embeddings represent the medical code descriptions.","marker":"Chen et al., 2024"},{"why":"Provides the MIMIC-IV dataset and its clinical code tables used for the main adult experiments.","marker":"Johnson et al., 2023"},{"why":"Provides the PIC pediatric intensive care dataset used to test generalization beyond adults.","marker":"Zeng et al., 2020"},{"why":"Defines Cehr_Bert, the strongest baseline that uses discrete artificial time tokens and time2vec embeddings.","marker":"Pang et al., 2021"},{"why":"Defines HiTANet, the hierarchical time-aware attention baseline that TALE-EHR compares against.","marker":"Li et al., 2022"},{"why":"Defines Retain and Doctor AI, recurrent baselines that TALE-EHR must beat on sequence and disease tasks.","marker":"Choi et al., 2016"},{"why":"Supplies the marked temporal point process framework underlying the timing loss and the conditional intensity formulation.","marker":"Daley and Vere-Jones, 2003"},{"why":"Defines the LSTM baseline used across the disease and code prediction comparisons.","marker":"Hochreiter and Schmidhuber, 1997"}],"fun_headline_variants":["Time gaps in attention improve EHR forecasting","Modeling irregular event times boosts EHR predictions","Time-aware attention beats baselines on MIMIC-IV","Explicit time gaps in attention improve EHR modeling","TALE-EHR: Learned time decay sharpens disease prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the disease-onset evaluation being built without temporal leakage: the paper says positive patients are scored on the 1024 events before their first diagnosis, but it never specifies how negative patients are defined, which index time they receive, or how censoring is handled, so if those choices differ between cases and controls the reported AUC gains could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Time gaps in attention improve EHR forecasting","Modeling irregular event times boosts EHR predictions","Time-aware attention beats baselines on MIMIC-IV","Explicit time gaps in attention improve EHR modeling","TALE-EHR: Learned time decay sharpens disease prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2372,"prompt_tokens":915,"completion_tokens":1457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1384}},"tokens_in":531,"tokens_out":1457,"duration_ms":12714,"temperature":1.0,"reasoning_tokens":1384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:46:27.698861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the nine disease tasks on MIMIC-IV with explicit negative-patient construction: each control is assigned the same index time as a matched case (or a random index from the control's own record) and the prediction window ends at that index; drop all cases with fewer than 1024 prior events exactly as the paper does. If TALE-EHR's AUROC advantage over the strongest baseline, Cehr_Bert, shrinks to within noise under this leakage-free labeling, then the reported superiority is an artifact of label construction rather than of time-aware attention.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-IV dataset and its clinical code tables used for the main adult experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PIC pediatric intensive care dataset used to test generalization beyond adults."},{"cited_title":"Jiang, K","cited_arxiv_id":null,"evidence_quote":"Defines Cehr_Bert, the strongest baseline that uses discrete artificial time tokens and time2vec embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the marked temporal point process framework underlying the timing loss and the conditional intensity formulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LSTM baseline used across the disease and code prediction comparisons."}],"review_version":1}