{"id":"c7863c7d-9542-49f0-8167-0b0061355825","arxiv_id":"2412.20040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TEMPT, a contrastive pretraining model with per-hospital prompt tuning, outperforms existing baselines on multi-center medication recommendation in the eICU dataset.","lead":"A new AI system called TEMPT pre-trains on patient diagnosis and procedure records from many hospitals, then adapts to each hospital with small learned prompt vectors, reporting better medication recommendation accuracy on the eICU dataset. The design specifically targets small hospitals with few records, which normally lack the data needed to train accurate models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8:1:1 split is only described at the record level; with eICU's multiple ICU stays per patient, cross-split patient overlap could inflate all reported gains, including the headline SOTA claim.","rationale":"I read the paper in good faith: the proposed two-stage pretrain-and-prompt-tune design is coherent and the ablated variants behave consistently. However, the strongest empirical claim — that TEMPT outperforms all baselines — depends critically on the evaluation split. The reader's weakest_assumption focused on heavily modified medication baselines. That is a real concern, but it is not the most load-bearing: even if GAMENet, COGNet, and G-Bert are dismissed as crippled, TEMPT still beats STAR, MMOE, and PLE, which are implemented much more faithfully. The evaluation split, by contrast, is the foundation of every comparison in the paper. If eICU patients contribute multiple records and those records straddle the train/test boundary, all reported metrics, including the internal ablations and the small-hospital gains, are potentially inflated. Because the paper never describes a patient-level split or reports patient counts, this gap must be closed before the SOTA claim can be accepted. I therefore keep the verdict CONDITIONAL, but attach an explicit patient-split condition that the current reader's rationale did not include.","tokens_in":26751,"tokens_out":6999,"duration_ms":75368,"concrete_test":"Inspect the released repository and re-run the pipeline with a patient-level split: group all records by patient identifier, assign whole patients to train/val/test in an 8:1:1 ratio per hospital, then rerun TEMPT, STAR, and Single-Train. If the TEMPT–STAR PRAUC gap shrinks below statistical significance or the ranking in Table 3 changes, the record-level split is inflating the results. Also report the fraction of patients with multiple records in the preprocessed cohort to establish the leakage potential.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that TEMPT significantly outperforms SOTA baselines on multi-center medication recommendation. All of that evidence rests on the train/validation/test protocol in §4.1.1: 'we divide the data into train/validation/test by the ratio of 8:1:1 for each hospital.' The unit is 'records', not patients, and the paper never states that records from the same patient are kept in the same split. eICU contains multiple ICU stays per patient, so a record-level split can place the same patient's stays in both training and test sets. Medication regimens are highly patient-specific and repeat across stays, so a model can memorize patient-level patterns instead of learning to generalize to new patients. This would inflate every metric in Tables 3–7 and could exaggerate TEMPT's advantage, especially since its pretraining and hospital-specific prompts may exploit repeated patients more than simpler baselines. The paper reports 102,363 records but no patient count, no explicit patient-split statement, and no leakage analysis. This concern is more load-bearing than the weakened GAMENet/COGNet baselines: TEMPT also beats the more faithfully implemented STAR, MMOE, and PLE, so the SOTA claim does not hinge on those medication baselines, but it hinges entirely on the split.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TEMPT is a two-stage model for multi-center (multi-hospital) medication recommendation. In stage 1, a shared transformer encoder is pretrained on diagnosis and procedure sets from all hospitals with a masked-token prediction loss and a contrastive alignment loss between diagnosis and procedure representations; the pretraining does not use medication labels. In stage 2, the encoder and code embeddings are frozen and, for each hospital, only a small set of prompt vectors and a medication-prediction MLP are trained. Experiments on eICU (102,363 records from 80 hospitals) compare TEMPT against medication recommendation baselines (Leap, GAMENet, G-Bert, COGNet), LLM baselines (GPT-4, TALLRec), multi-domain recommenders (MMOE, PLE, STAR), and TEMPT variants, using PRAUC, Jaccard, and F1. The paper reports statistically significant improvements over all baselines, with particularly large gains for small hospitals, plus ablations, PEFT comparisons, efficiency analysis, and hyperparameter sensitivity. Code is released.","tokens_in":27041,"tokens_out":7617,"duration_ms":70731,"significance":"If the reported results are valid, TEMPT makes a useful contribution: it identifies a realistic low-resource multi-center medication recommendation setting, shows that self-supervised pretraining on diagnosis and procedure co-occurrence transfers across hospitals, and demonstrates that per-hospital prompt tuning is more parameter-efficient than full finetuning. The paper is strong on reproducibility: code is public, five random splits and seeds are used, standard deviations and t-tests are reported, and the ablation and hyperparameter studies are reasonably comprehensive. The main caveat is that the headline SOTA claim rests on an evaluation protocol whose leakage risk is not addressed, a nonstandard PRAUC definition, and several heavily modified baselines; these issues affect the interpretation of nearly every table.","major_comments":[{"comment":"The train/validation/test split is described only at the level of records (we divide the data into train/validation/test by the ratio of 8:1:1 for each hospital), and Table 2 reports 102,363 records but no patient count. eICU contains multiple ICU stays per patient, so a record-level split can place the same patient's records in both training and test sets. Because medication regimens are strongly patient-specific and repeat across stays, this can inflate all metrics in Tables 3-7 and may exaggerate TEMPT's advantage, especially given its pretraining and hospital-specific prompts. Please re-run the experiments with a patient-level split (or otherwise demonstrate that patients do not cross splits) and report the patient counts per split.","section":"Section 4.1.1, Table 2"},{"comment":"The PRAUC definition is not the standard area under the precision-recall curve. Eq. (17) sums Precision_i times Recall_i over k=1 to |M| without defining a threshold index or a change in recall, whereas PR-AUC normally integrates Precision as a function of Recall (for example, average precision with (Recall_k - Recall_{k-1}) times Precision_k). Since PRAUC is the headline metric in Tables 3, 4, 5, and 7, please clarify the exact computation, including how thresholds are generated, and if the intention is a different metric, rename it and justify the choice.","section":"Section 4.1.4, Eq. (17)"},{"comment":"The claim that TEMPT outperforms state-of-the-art medication recommendation models is weakened by the baseline adaptations: GAMENet is run without its DDI graph and dynamic memory module, COGNet without its copy module, and G-Bert with procedures substituted for medications. These are central components of the original methods, so the comparisons are against substantially altered versions. TEMPT's advantage over MMOE, PLE, and STAR is less affected, but the state-of-the-art medication recommendation wording should be revised, or faithful implementations should be provided.","section":"Section 4.1.2, Table 3"},{"comment":"In the contrastive loss, the denominator sums over j not equal to i only and omits the positive pair exp(sim(u_i^d, u_i^p)/tau). Standard InfoNCE-style losses include the positive in the denominator. Please clarify whether this omission is intentional and how it affects the optimization; if it is a typo, correct the equation and re-run the pretraining experiments.","section":"Section 3.3.2, Eq. (9)"}],"minor_comments":[{"comment":"Algorithm 1 updates only E_d, E_p, and Theta_encoder, but the pretraining objective in Eq. (11) also contains the mask-prediction MLPs and the contrastive projectors; please list all updated parameters.","section":"Algorithm 1, line 6"},{"comment":"The inference paragraph says the recommending probability is given as Equation (12) shows, but Eq. (12) is the non-prompted finetuning formulation; inference should use the prompted representation in Eq. (14).","section":"Section 3.5"},{"comment":"Table 1 lists E_p^h as prompt embedding matrices of diagnosis and medication; medication should be procedure.","section":"Table 1"},{"comment":"Table 7 introduces TEMPT (finetune-freeze) without defining it in Section 4.7 or in the implementation details; please clarify what this variant is.","section":"Table 7, Section 4.7"},{"comment":"The Figure 7 caption says randomly sampled 8 hospitals, but the figure contains nine panels labeled (a) through (i).","section":"Figure 7"},{"comment":"The text says the reported metrics are averaged over all of the records and hospitals; please specify whether the averaging is per record then per hospital or per record pooled across hospitals, since the two procedures weight hospitals differently.","section":"Section 4.1.4"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is the most serious issue: if the authors cannot re-run the experiments with a patient-level split, the central SOTA claim is not supportable. The PRAUC definition also needs to be aligned with standard usage before the numerical comparisons can be interpreted. The rest of the manuscript is well-executed in terms of reproducibility and ablation coverage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent paper on a genuinely new task formulation, but the evaluation has a load-bearing vulnerability the authors don't address. The record-level 8:1:1 split in eICU likely mixes ICU stays from the same patient across train and test. Medication patterns are patient-specific, so that could inflate all the headline numbers. If I were refereeing, the first thing I'd ask for is a patient-level split or at least a leakage analysis.\n\nWhat's genuinely new: the multi-center medication recommendation setting itself, and the two-stage design that combines masked prediction and contrastive alignment of diagnosis/procedure sets with hospital-specific input prompts. That's a reasonable application of known building blocks, not a conceptual breakthrough. The experimental work is unusually careful: ablations, five seeds, significance tests, code on GitHub, a storage/computation analysis, and a breakdown by hospital size. The small-hospital improvement over Single-Train (9.2% PRAUC) is the most practically interesting result.\n\nThe soft spots, in proportion: first, the split issue above is the big one. The paper says 'records' are split, never mentions patients, and reports no patient count. eICU has multiple stays per patient. This isn't a fatal flaw proven; it's an unaddressed risk that could change the main conclusion. Second, the PRAUC definition in Eq. (17) is not the standard precision-recall AUC; it's a sum over medications of precision times recall. That's a red flag for metric comparability. Third, the heavily modified baselines (GAMENet without DDI/memory, COGNet without copy, G-Bert with procedures as medications) undercut the 'better than SOTA medication recommendation' claim, though the fact that TEMPT also beats STAR/MMOE/PLE, which are implemented faithfully, shores up the general conclusion. Fourth, the LLM baselines are single runs and are really there for scale, not as serious competitors. The paper's self-acknowledged limitation about ignoring DDI is honest but important for any clinical use.\n\nWho's this for? Anyone working on multi-center EHR modeling or hospital-specific adaptation. It deserves a serious referee, but the authors should be asked to fix the split, clarify the metric, and re-run with patient-wise splitting before acceptance.","headline":"Solid empirical paper on a new multi-center medication recommendation setting, but the record-level split risks patient leakage and needs to be fixed before the SOTA claim is credible.","tokens_in":27549,"tokens_out":2862,"would_cite":false,"duration_ms":30280,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A shared pretrained medical encoder plus a small learnable prompt per hospital can serve many hospitals at once, with the largest accuracy gains where data is scarce.","keywords":["medication recommendation","multi-center learning","electronic health records","contrastive pretraining","prompt tuning","transformer encoder","eICU dataset","catastrophic forgetting"],"falsifier":"Re-run the comparison with the original baseline components intact, giving GAMENet its DDI graph and dynamic memory, COGNet its copy module, and G-Bert its historical medications, and check whether each still trails TEMPT on eICU. If any of these full baselines matches or exceeds TEMPT's PRAUC of 0.5468, the reported superiority would be an artifact of weakened competitors rather than a property of TEMPT's design.","tokens_in":26574,"feed_emoji":"💊","tokens_out":7313,"duration_ms":63017,"temperature":0.7,"pith_summary":"This paper aims to show that medication recommendation can work across many hospitals at once, even when most hospitals have few records. It proposes TEMPT, a two-stage model: first pretrain a transformer encoder on diagnosis and procedure codes from all hospitals using masked-code prediction and a contrastive task that aligns diagnosis and procedure representations; then, for each hospital, freeze the pretrained weights and learn only a small hospital-specific prompt vector plus a prediction head. The authors claim this beats standard medication-recommendation models, multi-domain recommendation models, LLM baselines, and full finetuning on the eICU dataset, and that the gain is largest for small hospitals. A sympathetic reader should care because most hospitals lack enough data to train their own recommender, and the paper offers a way to share medical knowledge across hospitals without overwriting it.","feed_headline":"Small hospital prompts lift drug picks across 80 sites","feed_subtitle":"Shared pretraining plus tiny per-hospital prompts beats training each hospital alone; small hospitals gain the most.","key_machinery":"The load-bearing mechanism is a two-stage training scheme built on a shared transformer medical encoder. In pretraining, a mask prediction task randomly masks diagnosis and procedure codes and reconstructs them, capturing intra-set co-occurrence, while a contrastive task aligns the diagnosis representation and procedure representation of the same record against negatives from other records, capturing inter-set relationships. In the second stage, each hospital gets a small set of learnable prompt embeddings inserted at the start of the diagnosis and procedure sequences; the pretrained embedding matrices and encoder are frozen, and only the prompts and a medication-prediction MLP are updated. The prompt vector is what carries the hospital-specific adaptation without modifying the shared medical knowledge.","core_discovery":"On the paper's own terms, the discovery is that hospital heterogeneity in prescribing can be handled by separating general medical knowledge from hospital-specific adaptation: pretrain on all hospitals with two self-supervised tasks, then adapt per hospital with continuous prompts rather than finetuning all parameters. The empirical claim is that TEMPT reaches PRAUC 0.5468, Jaccard 0.3318, and F1-score 0.4790 on eICU, with statistically significant improvements over the best baseline, and improves small-hospital PRAUC by 9.20% over Single-Train. The authors also claim the prompt-tuning stage is cheaper in both training time and storage than full finetuning and that it relieves catastrophic forgetting.","pith_inferences":["The same pretrain-and-prompt recipe could carry over to other multi-site clinical prediction tasks, such as predicting in-hospital mortality or length of stay, where site differences are known to hurt a single global model.","Because the prompt embeddings are indexed by hospital, TEMPT opens a natural path to federated learning: hospitals could update only their own prompts locally and share the pretrained encoder without transferring raw patient data.","The paper's own stated limitation, ignoring drug-drug interactions, suggests a testable extension: add a DDI-constrained loss at the recommendation head and measure whether the multi-center gains survive when safety is enforced.","A direct testable implication is that the contrastive task's benefit should grow with distribution shift; comparing TEMPT's margin over pretraining without the contrastive task on hospital pairs with high versus low Jensen-Shannon divergence would isolate that effect."],"forward_implications":["If TEMPT's result holds, hospitals with fewer than 1,000 records gain the most from multi-center pretraining, with a 9.20% PRAUC improvement over training on their own data.","Freezing the shared medical encoder and learning only small prompts means the per-hospital adaptation cost is a small fraction of full finetuning, making deployment across dozens of hospitals feasible.","The two self-supervised pretraining tasks each contribute to the result; removing mask prediction or the contrastive task lowers performance, so learning relationships among diagnosis and procedure codes is part of the recipe.","The result suggests that a unified pretrained medication model can serve many hospitals, challenging the assumption that each hospital needs its own recommender trained from scratch."],"supporting_citations":[{"why":"Supplies the only public multi-center critical-care dataset used in all experiments, with records from 80 hospitals.","marker":"[53]"},{"why":"The pretraining-based medication recommendation baseline, adapted by substituting procedures for medications.","marker":"[60]"},{"why":"The memory-and-graph medication recommendation baseline, run without the DDI graph and dynamic memory module.","marker":"[61]"},{"why":"The longitudinal medication recommendation baseline, run without its copy module.","marker":"[82]"},{"why":"The instance-based medication recommendation baseline that takes only diagnosis as input.","marker":"[87]"},{"why":"The multi-task multi-domain baseline using shared expert networks with gating.","marker":"[51]"},{"why":"The multi-domain baseline separating specific and shared experts.","marker":"[67]"},{"why":"The multi-domain CTR baseline with star-topology domain-specific networks.","marker":"[63]"},{"why":"Source of the continuous prompt idea adapted into hospital-specific prompt tuning.","marker":"[36]"},{"why":"Shows learnable prompt vectors work for tuning pretrained vision models and is cited as inspiration.","marker":"[24]"}],"fun_headline_variants":["Per-hospital prompts beat finetuning for small clinics","Small hospitals gain most from contrastive pretrain and prompt tuning","Contrastive pretraining plus prompts outperform single-hospital models","Prompt tuning tailors medication AI to each hospital's habits","TEMPT model separates shared knowledge from per-site prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison against prior methods assumes the re-implemented baselines are faithful to their original designs, but several were run in stripped-down form: GAMENet without its drug-drug interaction graph and memory module, COGNet without its copy module, and G-Bert with procedures substituted for medications.","fun_headline_variants_meta":{"raw":{"variants":["Per-hospital prompts beat finetuning for small clinics","Small hospitals gain most from contrastive pretrain and prompt tuning","Contrastive pretraining plus prompts outperform single-hospital models","Prompt tuning tailors medication AI to each hospital's habits","TEMPT model separates shared knowledge from per-site prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3335,"prompt_tokens":990,"completion_tokens":2345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2265}},"tokens_in":606,"tokens_out":2345,"duration_ms":18740,"temperature":1.0,"reasoning_tokens":2265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:37:24.393536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with the original baseline components intact, giving GAMENet its DDI graph and dynamic memory, COGNet its copy module, and G-Bert its historical medications, and check whether each still trails TEMPT on eICU. If any of these full baselines matches or exceeds TEMPT's PRAUC of 0.5468, the reported superiority would be an artifact of weakened competitors rather than a property of TEMPT's design.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The multi-domain baseline separating specific and shared experts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The multi-domain CTR baseline with star-topology domain-specific networks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The multi-task multi-domain baseline using shared expert networks with gating."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The pretraining-based medication recommendation baseline, adapted by substituting procedures for medications."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The memory-and-graph medication recommendation baseline, run without the DDI graph and dynamic memory module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The longitudinal medication recommendation baseline, run without its copy module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The instance-based medication recommendation baseline that takes only diagnosis as input."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the continuous prompt idea adapted into hospital-specific prompt tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows learnable prompt vectors work for tuning pretrained vision models and is cited as inspiration."}],"review_version":1}