{"id":"735c2ea5-cb6c-47b8-90eb-e08be36a5497","arxiv_id":"2506.11082","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A decoder-only transformer trained on tokenized MIMIC-IV chest pain trajectories predicts next clinical events with validation perplexity 2.01, against a random baseline only.","lead":"PRISM is a GPT-style model trained on structured hospital event records to predict the next diagnostic test, lab result, or diagnosis in a timeline. The paper shows it beats a uniform random guess on next-token prediction, but not compared with simpler baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation perplexity 2.01 has no nontrivial baseline, so the central claim of capturing complex clinical dependencies is unsupported.","rationale":"I read the paper as making a feasibility claim: a decoder-only transformer can model structured clinical event sequences, and its low perplexity demonstrates substantive predictive structure. The core evidence is Table I, where validation perplexity 2.01 is interpreted as 'removing more than 92% of the intrinsic sequence entropy' relative to a uniform 10,000-way guess. That comparison sets a trivial floor; in a highly templated clinical domain, even a bigram model can exploit event-type tags and repeated test panels to achieve low perplexity. The paper's own admission that no explicit baselines were implemented means the strongest empirical claim is unverified. This is load-bearing because it attacks the central claim's evidential basis, not an ancillary detail. The reader's concern about deterministic ordering is valid as an interpretational confound, but the absence of baselines is more decisive: even if all timestamps were perfectly ordered, the results would still not show complex dependency capture. The proposed trigram/LSTM comparison would settle whether the transformer's long-range architecture adds value over local statistics. Since the reader's verdict was already CONDITIONAL and called for stronger evaluation, my read does not change that verdict; it sharpens the rationale. The paper is transparent about missing baselines and limitations, so there is no internal inconsistency or misrepresentation, but the central claim remains unsupported as currently evidenced.","tokens_in":7686,"tokens_out":5145,"duration_ms":50257,"concrete_test":"Train a Kneser-Ney smoothed trigram (or an LSTM of comparable capacity, e.g., embedding dimension 256, hidden dimension 256) on the exact same tokenized patient documents and the same 80/10/10 split, and compute validation perplexity under the same 10,000-token vocabulary. If the simple baseline attains a perplexity within about 10% of PRISM's 2.01, the claimed advantage over non-random models and the inference of complex dependency capture are not supported; the paper would need to show a large margin and/or a controlled long-range task (e.g., predicting events across admission boundaries or after more than 100 tokens) to substantiate its central claim. Also report the uniform-random perplexity on this token distribution for context.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim (Abstract; Section IV.B, Table I) is that PRISM 'substantially improves over random baselines' in next-token prediction, with validation perplexity 2.01. But the only comparator is a uniform 10,000-way guess (log2 10,000 = 13.3 bits). In a domain where the cohort is deliberately homogeneous (3,164 chest-pain-to-cardiac patients, Section III) and tokenization imposes a fixed event-category ordering (admission -> OMR -> lab -> microbiology -> diagnosis -> discharge, Section III), a uniform baseline is nearly meaningless: any model that learns even the event-type template or the most frequent tests will beat it by a large margin. The paper itself states 'no explicit baseline models (e.g., n-gram predictors or recurrent neural networks) were implemented' (Section IV.C). Without comparing to a trigram or LSTM baseline under the same tokenization and split, a perplexity of 2.01 cannot distinguish genuine long-range dependency learning from exploitation of local template structure. The subsequent claim that the model 'internalizes complex clinical reasoning patterns' rests on three hand-picked qualitative generations (Section IV), not on a quantitative advantage. The acknowledged artificial ordering of simultaneous events (Section III; Discussion limitation 3) further confounds the clinical interpretation, but the empirical issue is more fundamental: the evidence as reported cannot support the central claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PRISM, a decoder-only transformer trained on tokenized MIMIC-IV clinical event sequences for patients whose trajectories begin with unspecified chest pain and progress to a confirmed cardiac diagnosis. The tokenization pipeline converts six event classes (demographics, admissions/discharges, labs, outpatient measurements, microbiology, and ICD diagnoses) into a 10,000-token vocabulary, with events sorted chronologically and, for simultaneous events, by a fixed event-type and alphabetical ordering. The model is trained with a causal language modeling objective and evaluated by next-token perplexity, attaining a validation perplexity of 2.01. The authors interpret this as a substantial improvement over a uniform random baseline and support the claim with three qualitative next-token generation examples. The paper explicitly acknowledges the limitations of its narrow cohort, the exclusion of procedures and medications, and the artificial ordering imposed on co-occurring events.","tokens_in":7966,"tokens_out":4458,"duration_ms":52140,"significance":"If the central claims were supported, the paper would demonstrate a useful proof-of-concept for applying generative language modeling to structured EHR data, with potential applications in decision support, simulation, and education. The work is clearly motivated, uses a publicly available dataset, and provides a transparent description of its tokenization and training pipeline. However, the current evidence is insufficient to establish the main claim: the only quantitative comparison is a uniform baseline, the generation results are qualitative and hand-picked, and the deterministic ordering of simultaneous events confounds clinical interpretation. The manuscript would be of significantly greater value if the evaluation included standard sequence-modeling baselines, test-set metrics, and a systematic assessment of generated sequences.","major_comments":[{"comment":"The only quantitative comparator is a uniform distribution over the 10,000-token vocabulary. In a cohort that is already homogeneous (chest pain progressing to cardiac disease; Section III) and whose tokenization imposes a fixed event-type order, a model that learns even the event template or the most frequent tests can outperform uniform random by a wide margin. The paper itself states that no n-gram or recurrent baselines were implemented. Reporting perplexity 2.01 against uniform therefore cannot support the central claims of 'substantial improvements' or capturing 'complex dependencies across longitudinal patient timelines.' Please add at least a trigram/Kneser-Ney baseline, an LSTM, or a frequency-based template model trained on the same tokenization and split.","section":"Section IV.B (Table I) and Section IV.C"},{"comment":"The paper concedes that events sharing a timestamp were assigned a fixed deterministic order based on event type and alphabetical sorting, introducing an artificial ordinality. As a result, the next-token prediction task is partly a task of learning this sorting rule rather than genuine clinical decision-making. Please quantify this confound, for example by comparing against a permutation-invariant or set-based formulation (such as DPSS, cited as [10]), or by evaluating on shuffled orderings. The clinical-reasoning interpretation should be tempered unless this ordering effect is shown to be minor.","section":"Section III (Tokenization Strategy) and Discussion, limitation 3"},{"comment":"The three scenarios in Figure 4 are interpreted narratively, with no quantitative scoring, no comparison with baseline-generated sequences, no error analysis, and no clinician evaluation. These hand-picked examples do not support the statement that PRISM 'internalizes complex clinical reasoning patterns.' Please supplement with automatic metrics (e.g., top-k accuracy on held-out continuations, repetition rate, diversity) or blinded clinician ratings, and report results over a representative sample of prompts rather than a hand-selected subset.","section":"Section IV (Next-Token Generation Experiments)"},{"comment":"The data were split 80/10/10, but only validation metrics are reported, and the best checkpoint is selected by validation loss. Without test-set perplexity, confidence intervals, or subject-level variability estimates, the reported 2.01 validation perplexity cannot be assessed for statistical significance. Given the relatively small cohort (3,164 patients) and 10,000-token vocabulary, please report test-set performance with uncertainty estimates.","section":"Section III (Training Procedure) and Section IV.B"}],"minor_comments":[{"comment":"The phrase 'random baselines' is used without specifying what the baseline is; clarify that it is a uniform distribution over the vocabulary and add a non-uniform baseline.","section":"Abstract and Section IV.C"},{"comment":"There is a typo in the Figure 1 caption ('CARDIOV ASCULAR'), and the text '¿200 k' in Section II.C appears to be an encoding artifact that should be corrected.","section":"Figure 1 and Section II.C"},{"comment":"Several references are anonymous preprints or non-archival sources (e.g., [1] is a company blog and [2]-[5] are anonymous); consider replacing them with peer-reviewed publications or providing full author and venue information.","section":"References"},{"comment":"The text attributes the training/validation loss gap to 'dropout and weight decay,' but the Training Procedure section does not report dropout or weight-decay hyperparameters; please specify them.","section":"Section IV.A"},{"comment":"The statement that 'temporal ordering of events was verified using timestamped diagnosis records' should include the verification rule and how timestamp ties or inconsistencies were handled.","section":"Section III, Patient Selection"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about its limitations and the tokenization pipeline is clearly described, but the evaluation section is not yet at the standard required for the claimed conclusions. The missing baseline comparison, test-set metrics, and systematic generation evaluation are the key gaps. These are addressable within the scope of a revision, so I do not recommend rejection, but the central claim cannot be accepted as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take on arXiv:2506.11082.\n\nThe honest summary: this is a competent feasibility study. The authors build a tokenization pipeline for six MIMIC-IV event classes, construct a chest-pain-to-cardiac cohort (3,164 patients), train a small GPT-2-style decoder, and report validation perplexity 2.01. The methods are clearly described, and the limitations section is unusually candid — they explicitly note the artificial ordering forced on simultaneous events and the lack of baselines. That candor is real and should count in the paper's favor.\n\nWhat's new is narrow: a specific tokenization strategy and a new cohort application. The architecture is standard, and as the authors acknowledge, ETHOS and TransformEHR already do transformer-on-EHR. So novelty is incremental.\n\nThe soft spot is exactly the one the stress test flags: the only comparator is a uniform 10,000-way guess. In a homogeneous cohort with a fixed event-type template (admission → OMR → lab → micro → diagnosis → discharge), any model that learns the template and common lab names will crush a uniform baseline. A trigram or LSTM under the same tokenization and split is the minimum needed to know whether the transformer is learning long-range dependencies or local template structure. Without that, \"substantially improves over random baselines\" is a true but nearly meaningless statement. The claim about \"complex clinical reasoning patterns\" rests on three hand-picked generations.\n\nOther soft spots: no code or data released (only the MIMIC-IV public data, but no tokenizer or model), so the results are hard to reproduce independently; and the generation evaluation is qualitative with no quantitative scoring. The arbitrary ordering of simultaneous events is a real confound, but the authors already flag it. The cohort filtering is at least clearly described.\n\nIs the central argument sound? Not as stated. The evidence supports \"a GPT-style model can achieve low perplexity on this tokenized clinical corpus,\" not \"captures complex dependencies across longitudinal patient timelines.\" But the gap is fixable: add proper baselines, release artifacts, and temper the claims.\n\nMy recommendation: this deserves a serious referee, because it's a reasonable feasibility demonstration and the limitations are honestly stated. I'd send it out, but with the expectation of heavy revision — the baseline comparison is mandatory, not optional.\n\nI wouldn't cite it in my own work until the artifacts and baselines appear.","headline":"Validation perplexity 2.01 with only a uniform-random baseline cannot support the claim of capturing complex clinical dependencies, but the paper is an honest feasibility study that deserves refereeing with mandatory baseline comparisons.","tokens_in":8480,"tokens_out":2148,"would_cite":false,"duration_ms":20542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a small decoder-only transformer can learn the next step in a patient's cardiac diagnostic timeline from tokenized clinical events, reaching validation perplexity 2.01 on a 10,000-token vocabulary.","keywords":["PRISM","clinical decision support","next-token prediction","decoder-only transformer","electronic health records","diagnostic workflows","tokenization","MIMIC-IV"],"falsifier":"Re-tokenize the same 3,164-patient cohort with simultaneous events given random or permuted orderings, retrain the identical architecture, and compare validation perplexity and generated sequences; if perplexity remains near 2.01 with equally plausible outputs, the model is learning the sorting artifact, not latent clinical reasoning. A complementary check is fitting a simple n-gram or Markov baseline on the same tokens; if it achieves comparable perplexity, the transformer-specific dependency claim is not established.","tokens_in":7514,"feed_emoji":"🩺","tokens_out":7622,"duration_ms":73381,"temperature":0.7,"pith_summary":"The paper claims that clinical diagnostic workflows can be modeled as a language: after converting a patient's tests, results, and diagnoses into a chronological sequence of tokens, an autoregressive transformer learns to predict the next clinical event with a validation perplexity of about 2.01 on a 10,000-token vocabulary. The authors argue that this is far better than the uniform random baseline and shows that structured electronic health record data contain learnable, recurring patterns of clinician decision-making. If true, the approach could support decision-support systems that suggest next diagnostic steps, flag departures from standard care pathways, and generate realistic patient trajectories for training and simulation.","feed_headline":"A transformer predicts next clinical events at perplexity 2.01","feed_subtitle":"Trained on real hospital cardiac workups, PRISM narrows a 10,000-token vocabulary to about two likely next steps.","key_machinery":"The central mechanism is the tokenized patient timeline: a deterministic, chronological serialization of clinical events into text-like tokens, produced by a batch pipeline that relabels event entries, sorts them by timestamp, and breaks ties with a fixed event precedence (admission to outpatient measurement to lab to microbiology to diagnosis to discharge), followed by alphabetical sorting within each event type. This converts the clinical record into a sequence that a causal transformer can learn on, so that the language-modeling objective — predict the next token — becomes the model of clinical decision-making. The architecture is a small decoder-only transformer with learned positional embeddings, a 10,000-token frequency-pruned vocabulary with reserved padding and unknown tokens, and a cross-entropy training objective.","core_discovery":"PRISM is a GPT-2-inspired decoder-only transformer with 6 layers, 8 attention heads, and 256-dimensional embeddings, trained with a causal language-modeling objective on tokenized timelines of 3,164 public electronic health record patients who presented with unspecified chest pain and later received a confirmed cardiac diagnosis. The tokenization pipeline converts six classes of events — demographics, admissions and discharges, laboratory results, outpatient measurements, microbiology cultures, and ICD-coded diagnoses — into text-like tokens, sorted first by timestamp and then by a fixed event precedence and alphabetical order. On a held-out validation split, the best checkpoint reaches cross-entropy 0.7000, corresponding to a perplexity of 2.01, which the authors interpret as narrowing a uniform 10,000-way choice to roughly two plausible next events and removing more than 92 percent of the sequence entropy. The paper also shows qualitative next-token generation examples in which the model emits clinically coherent continuations, such as serial cardiac biomarker testing after suspected acute coronary syndrome and a nephrology workup after elevated creatinine.","pith_inferences":["The paper does not implement an n-gram, Markov, or recurrent baseline, so its implicit claim that a transformer is specifically needed for these sequences remains untested; a simple local baseline on the same tokens could plausibly match perplexity around 2 because clinical protocols are highly local.","Because the deterministic tie-breaking sorts simultaneous events alphabetically, part of the predictive signal may come from the sorting rule itself; retraining with random tie orders would quantify how much of the reported perplexity reflects the ordering artifact the paper concedes in its third limitation.","The cohort is restricted to patients who began with unspecified chest pain and ended with a cardiac diagnosis, so the claim about capturing complex longitudinal dependencies is demonstrated only within one narrow, protocol-heavy pathway; extending to diverse presentations and non-cardiac outcomes is an open question.","If the next-token probabilities are shown to remain stable under perturbed event orderings, the model could be used not only to suggest next tests but to identify which historical events actually drive each suggestion, giving an explanation layer through attention weights."],"forward_implications":["Next-token probabilities can be used to rank candidate next diagnostic actions, enabling a top-k clinical decision-support list for chest-pain and cardiac workups.","The trained model can generate synthetic patient trajectories that follow standard care pathways, providing material for medical education and for stress-testing downstream decision-support pipelines.","The tokenized timeline representation yields a dense, contextual embedding of a patient's state, which can be reused for risk stratification, anomaly detection, or cohort comparison beyond next-token prediction.","A validation perplexity of about 2 shows that much of the routine diagnostic workflow is regular and protocol-driven, so large deviations from predicted sequences can serve as flags for unusual or potentially unsafe care."],"supporting_citations":[{"why":"Supplies the public electronic health record data (MIMIC-IV) from which the chest-pain-to-cardiac training cohort is drawn.","marker":"[7]"},{"why":"Introduces the patient-health-timeline tokenization of events into sequential tokens that PRISM adapts and extends with its fixed tie-breaking order.","marker":"[2]"},{"why":"Provides the canonical sequence-of-sets alternative that represents simultaneous events as order-invariant sets, which PRISM contrasts with its deterministic serialization.","marker":"[10]"},{"why":"Demonstrates tokenizing standardized medical codes into a compact vocabulary for transformer training, informing the frequency-pruned vocabulary design.","marker":"[5]"},{"why":"Provides the expert-rule and Bayesian diagnostic aid whose limited disease coverage and brittle run-time behavior motivate the data-driven sequence model.","marker":"[6]"}],"fun_headline_variants":["PRISM: transformer predicts clinical events at perplexity 2.01","AI model narrows patient timeline prediction to perplexity 2.01","Leveraging GPT-style model for clinical sequences: perplexity 2.01","Chest-pain AI predicts next diagnostic steps with perplexity 2.01","PRISM: language model for structured medical data achieves perplexity 2.01"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fixed, alphabetical tie-breaking order assigned to events recorded at the same timestamp reflects the true sequence of clinical decisions; if the real order is not sequential, the next-token task is partly learning the sorting rule rather than clinical reasoning.","fun_headline_variants_meta":{"raw":{"variants":["PRISM: transformer predicts clinical events at perplexity 2.01","AI model narrows patient timeline prediction to perplexity 2.01","Leveraging GPT-style model for clinical sequences: perplexity 2.01","Chest-pain AI predicts next diagnostic steps with perplexity 2.01","PRISM: language model for structured medical data achieves perplexity 2.01"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":2968,"prompt_tokens":929,"completion_tokens":2039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1937}},"tokens_in":545,"tokens_out":2039,"duration_ms":17163,"temperature":1.0,"reasoning_tokens":1937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:55:55.545483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-tokenize the same 3,164-patient cohort with simultaneous events given random or permuted orderings, retrain the identical architecture, and compare validation perplexity and generated sequences; if perplexity remains near 2.01 with equally plausible outputs, the model is learning the sorting artifact, not latent clinical reasoning. A complementary check is fitting a simple n-gram or Markov baseline on the same tokens; if it achieves comparable perplexity, the transformer-specific dependency claim is not established.","supporting_citations":[{"cited_title":"A Transformer-Based Model for Zero- Shot Health Trajectory Prediction","cited_arxiv_id":null,"evidence_quote":"Introduces the patient-health-timeline tokenization of events into sequential tokens that PRISM adapts and extends with its fixed tie-breaking order."},{"cited_title":"Diag- nostic Prediction with Sequence-of-Sets Representation Learning for Clinical Events","cited_arxiv_id":null,"evidence_quote":"Provides the canonical sequence-of-sets alternative that represents simultaneous events as order-invariant sets, which PRISM contrasts with its deterministic serialization."},{"cited_title":"Transformer-Based Deep Learning Model for the Diagnosis of Lung Cancer in Primary Care","cited_arxiv_id":null,"evidence_quote":"Demonstrates tokenizing standardized medical codes into a compact vocabulary for transformer training, informing the frequency-pruned vocabulary design."},{"cited_title":"Variational probabilistic inference and the QMR-DT network","cited_arxiv_id":null,"evidence_quote":"Provides the expert-rule and Bayesian diagnostic aid whose limited disease coverage and brittle run-time behavior motivate the data-driven sequence model."}],"review_version":1}