{"id":"9f5958ad-4251-46cc-9ae0-828318daef18","arxiv_id":"2501.18724","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Open-source LLMs still struggle with temporal reasoning in long clinical summaries, and adding prior context or retrieval only partially helps.","lead":"This paper benchmarks five open-source large language models on summarizing long, multi-day electronic health records and on predicting future diagnoses. It finds that longer context windows and retrieval augmentation help only partly, while temporal reasoning and rare-disease prediction remain weak.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central temporal-reasoning claim is not directly measured: Section 5 uses only ROUGE-L, BERTScore, and F1, none of which isolate temporal ordering, so 'models struggle with temporal reasoning' is an inference that needs a direct temporal-consistency check.","rationale":"I considered the reader's weakest assumption about discharge summarization being limited to the last 24/48 hours (Section 3.1). That is a real limitation: it weakens the 'full patient trajectory' framing for that task and should be acknowledged. However, A&P generation with Multi-Day Context and EHRShot forecasting are genuinely longitudinal, so the truncation does not by itself overturn the central claim. The more load-bearing assumption is that the reported metrics constitute evidence about temporal reasoning. If that assumption fails, the paper's unique contribution—the temporal-reasoning component—is unsubstantiated even though weaker claims ('long-context summarization is hard', 'RAG helps somewhat') may still hold. The paper does have some independent support: the expert review in Figure 7 and Appendix A.11 points toward real clinical deficiencies such as outdated diagnoses persisting, and the authors transparently report limitations. Those qualitative signals make the overall direction plausible. But the quantitative case for a specifically temporal failure needs a direct probe, because ROUGE-L/BERTScore and binary F1 are not diagnostic of ordering or causal reasoning. This is an addressable measurement gap rather than a fatal flaw, so the reader's CONDITIONAL verdict remains appropriate; I would add the direct temporal-consistency evaluation as a condition for the strongest version of the claim.","tokens_in":20695,"tokens_out":5863,"duration_ms":58954,"concrete_test":"Select roughly 100 patient-days from the MIMIC A&P setup and extract gold event pairs with a clinically required temporal/causal order (e.g., an abnormal lab value precedes the therapy started in response; a medication precedes the symptom change attributed to it). Define a deterministic temporal-order accuracy score: does the model's generated Assessment/Plan mention both events and put them in the clinically required order? Independently, shuffle timestamps in the model input and re-generate. If shuffled inputs do not degrade temporal-order accuracy and chronological inputs yield near-chance order accuracy, the temporal-reasoning claim is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline conclusion—'Current models struggle with accurate summarization and temporal reasoning' (Section 7)—rests on aggregate metrics that cannot separate the two. Section 4.2 defines evaluation as ROUGE-L, SapBERT-BERTScore, and macro F1/accuracy. These reward token/semantic overlap with a single gold reference, not correct temporal ordering; a summary can be clinically accurate and temporally coherent yet score low, and a temporally jumbled summary can score well if it contains the right phrases. The only temporal control is the SHUFFLED TABULAR condition in discharge summarization (Figure 5), but it is applied to the final 24/48 hours, a window the authors themselves argue is clinically stable, and it tests whether order matters for that snapshot, not whether models can reason across a trajectory. In A&P generation, Single-Day vs. Multi-Day context differences (Table 4) show sensitivity to input length, not whether events are ordered correctly. CoT event extraction (Table 5) produces event lists, but their temporal correctness is never scored. On EHRShot (Table 6), Qwen's F1=0 on Celiac and Lupus with >95% negative prevalence is the expected majority-class outcome and demonstrates class-imbalance failure, not specifically temporal-reasoning failure. The distinctive claim of the paper therefore outruns the measurements; the results are consistent with generic summarization deficits and imbalance artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates five open-source LLMs (Mistral-7B, Llama3-8B, Qwen2.5-7B, DeepSeek-R1-Distill-Qwen-32B, Llama2-13B), their RAG variants, and a chain-of-thought event-extraction pipeline on two public EHR datasets (MIMIC-III and EHRShot), across three tasks: discharge summarization, assessment-and-plan (A&P) generation, and diagnosis prediction. It re-engineers these tasks into long-context, multi-modal settings with temporally ordered inputs, and reports ROUGE-L, SapBERT-BERTScore, and macro F1/accuracy, plus a small expert review. The main conclusions are that long context windows improve input integration but do not consistently enhance clinical reasoning, that models struggle with temporal progression and rare disease prediction, and that RAG yields only partial, inconsistent improvements.","tokens_in":21012,"tokens_out":4521,"duration_ms":41954,"significance":"If the findings hold, the paper provides a useful, reproducible evaluation of open-source LLMs on longitudinal clinical summarization and prediction, and it highlights a real gap: raw context length and retrieval do not by themselves solve temporal reasoning or rare-disease prediction. The design is methodical, uses public datasets, evaluates zero-shot on held-out patients, and compares multiple models, prompting strategies, and input modalities. The authors also provide a transparent account of preprocessing and hyperparameter selection. However, the paper's distinctive claim about temporal reasoning is not directly supported by the reported metrics, because none of ROUGE-L, BERTScore, or macro F1 isolates temporal ordering correctness. The manuscript is therefore a solid empirical study with an overclaimed interpretive layer, and it is best published after either adding direct temporal-consistency evaluation or rephrasing the conclusions to match what the metrics can actually show.","major_comments":[{"comment":"The headline conclusion that models \"struggle with ... temporal reasoning\" is not directly measured. ROUGE-L, SapBERT-BERTScore, and macro F1/accuracy reward lexical or semantic overlap with a gold reference or binary classification accuracy; they do not score whether generated events appear in the correct temporal order or whether causal links across time are preserved. The only temporal manipulation, SHUFFLED TABULAR in §3.1 and Figure 5, is applied to the final 24/48 hours, which the authors themselves describe as a clinically stable window, so it tests order sensitivity in a snapshot rather than reasoning across a trajectory. The extracted event lists in the CoT approach (Table 5) are never checked for temporal correctness. Please add a direct temporal-consistency evaluation (for example, event-order accuracy, timestamp-violation rate, or human judgments of temporal coherence) or restrict the conclusions to summarization and prediction deficits without the temporal-reasoning framing.","section":"Section 4.2 and Section 5 (Tables 2, 4, 5, 6)"},{"comment":"The discharge summarization task limits input to the last 24 or 48 hours of hospitalization, yet the abstract and introduction claim the paper evaluates \"full patient trajectories\" and \"long-context\" clinical data. With this window, any clinically important events that occurred earlier in the admission are never seen by the model, so the task measures short-window summarization rather than reasoning over the complete longitudinal course. This is a load-bearing mismatch between the task design and the claimed scope. Please either extend the input to the full stay (with appropriate length controls or retrieval over the full stay) or explicitly describe the discharge task as a short-window summarization task and adjust the abstract and conclusion accordingly.","section":"Section 3.1 (Discharge Summarization)"},{"comment":"RAG hyperparameters (chunk size, top-k, overlap) were selected on a held-out set of n=5, and the CoT temporal-context settings were selected on n=20 development patients, but no significance testing or confidence intervals are reported anywhere in the paper. Many comparative claims in Section 5 (for example, \"RAG generally improves performance\" and \"long context windows improve input integration\") are based on differences that fall within one standard deviation of the reported means, such as Mistral's discharge-summarization RAG vs. direct-generation scores (15.04 vs. 16.28 with σ≈2–13). Without significance tests or an analysis of how the small tuning sets affect generalization, these comparative conclusions are not established. Please add appropriate statistical tests or confidence intervals and discuss the sensitivity of the conclusions to the tuning procedure.","section":"Section 4.1 and Appendix A.4"},{"comment":"The near-zero F1 scores for Celiac and Lupus with accuracies around 95% are exactly what one expects from majority-class prediction given negative prevalence of 96.54% and 95.56%. These results demonstrate class-imbalance failure, not specifically a temporal-reasoning failure. The text in Section 5 partially acknowledges this, but the discussion should report balanced metrics such as AUROC or AUPRC, and should avoid presenting these results as evidence about temporal reasoning. Without such metrics, the reader cannot distinguish a model that ignores all temporal information from one that reasons well but is not calibrated for rare outcomes.","section":"Table 6 (EHRShot diagnosis prediction)"}],"minor_comments":[{"comment":"The word \"summarizatoin\" is a typo and should read \"summarization.\"","section":"Section 4.2"},{"comment":"In the Qwen row, the Single-Day ROUGE-L value is reported as 77.50 ± 5.37, which appears inconsistent with the value 20.74 reported in Table 4 for the same condition; this is likely a copy-and-paste error and should be corrected.","section":"Appendix A.9, Table 18"},{"comment":"The caption states \"Average f1 results across modalities and time windows,\" but the surrounding text discusses ROUGE-L and BERTScore; please clarify which metric (or average of metrics) is plotted.","section":"Figure 5 caption"},{"comment":"The phrase \"shuffled the tabular data by their timestamps\" is ambiguous; it should read \"shuffled the tabular data with respect to their timestamps\" or \"shuffled the timestamp order of the tabular data.\"","section":"Section 3.1"},{"comment":"The expert error analysis is based on 10 pairs of outputs from a single physician using an instrument that the authors explicitly state was not validated. This is an acknowledged limitation, but the claim in Section 6 that \"RAG performs slightly better, with fewer hallucinations\" should be presented more cautiously, for example as an exploratory observation.","section":"Section 6 and Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a serious empirical study with a clear task design and reproducible public-data evaluation, but the central temporal-reasoning claim is not backed by the reported metrics. The authors can address this either by adding a direct temporal-coherence evaluation or by scaling back the interpretive claims. The lack of significance testing and the very small tuning sets are also substantive concerns that should be fixed in revision. I recommend major revision rather than rejection, because the core contribution—a systematic evaluation of open LLMs on long-context clinical summarization—is valuable and the identified problems are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things before reading this: it is a useful, honest evaluation of open-source LLMs on long clinical sequences, and its headline claim about temporal reasoning is not backed by the metrics they report. Worth engaging, but the conclusion needs reining in.\n\nWhat's new: they take two public datasets (MIMIC-III, EHRShot) and re-engineer discharge summarization, A&P generation, and diagnosis prediction into long-context, multi-modal, timestamped tasks. They compare direct generation, RAG, and a CoT event-extraction step across five open models. That setup hasn't been done before, and the results are a useful caution: longer context windows and retrieval don't fix clinical reasoning, and rare-disease prediction collapses to majority-class behavior. The negative results are the contribution. They also report compute costs and include a small physician evaluation, which is a good-faith attempt at qualitative grounding.\n\nWhere it's soft: the central claim that models struggle with 'temporal reasoning' rests on ROUGE-L, BERTScore, and F1, none of which isolate temporal order. The shuffled-tabular condition only covers the final 24-48 hours, which they themselves call clinically stable, so it doesn't test trajectory reasoning. CoT event extraction never scores whether the extracted event lists are temporally correct. The zero F1 on rare diseases in EHRShot is exactly what class imbalance produces, not evidence of a temporal deficit. Second, the statistics are thin: RAG hyperparameters were tuned on five patients, CoT contexts on twenty, and there's no significance testing, so the modest RAG gains could be noise. Third, the discharge summarization task only uses the last 24-48 hours, which undercuts the 'full trajectory' framing. These are addressable.\n\nMy take: a solid benchmark paper with an overreaching interpretation. If the authors add a direct temporal-consistency evaluation, report significance or confidence intervals, and temper the 'fills the gap' language, it becomes a useful reference for clinical summarization. I'd send it to review; the evaluation matrix and public-dataset setup deserve referee time. I wouldn't cite it in current form without the temporal check, but a corrected version would be citable.\n\nRecommendation: engage, but condition acceptance on the temporal metric and statistical rigor.","headline":"Useful evaluation of open LLMs on long clinical sequences, but the headline temporal-reasoning claim isn't actually measured; needs a direct temporal check and statistical rigor.","tokens_in":21549,"tokens_out":4350,"would_cite":false,"duration_ms":36595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs still cannot reliably summarise long patient trajectories or reason across time; retrieval augmentation helps only partly.","keywords":["clinical text summarization","temporal reasoning","electronic health records","large language models","retrieval-augmented generation","chain-of-thought prompting","discharge summarization","diagnosis prediction"],"falsifier":"Give a long-context LLM a full multi-day admission instead of the last 24 or 48 hours and compare its discharge summary against the same model restricted to the short window; if full-input summaries reach clinician-level completeness and accuracy, the paper's claim that models cannot reason over long longitudinal trajectories would be undermined. A simpler version: find one 72-hour-plus admission where an event on day 1 determines the discharge diagnosis, and check whether the model using only the final window misses it while the full-input model catches it.","tokens_in":20491,"feed_emoji":"🩺","tokens_out":5252,"duration_ms":44912,"temperature":0.7,"pith_summary":"The paper tries to establish where current open-source large language models stand on longitudinal clinical reasoning: given multi-day, multi-modal electronic health records, can they write discharge summaries, generate daily Assessment and Plan notes, and predict future diagnoses? Across MIMIC-III and EHRShot, the answer is no, not reliably. Longer context windows help models integrate more input but do not consistently improve clinical reasoning, and retrieval-augmented generation reduces some hallucinations without fixing temporal progression or rare-disease predictions. A clinician's qualitative review confirms the automated scores: summaries oversimplify, retain outdated diagnoses, and can drift from the record. The intended contribution is an evaluation design that reframes clinical summarization as a long-context, temporally ordered task rather than a single-snapshot benchmark.","feed_headline":"LLMs still fail at temporal clinical reasoning","feed_subtitle":"Open-source LLMs miss rare diagnoses and struggle with longitudinal summaries even with RAG.","key_machinery":"The load-bearing mechanism is the task itself: each record is verbalized into natural language with relative timestamps, grouped by time, and either fed whole (direct generation), split and retrieved (RAG), or first compressed into a temporally ordered 'key events' list by chain-of-thought prompting before summarization. The comparison manipulates context availability: no prior context, single-day context, or multi-day context for progress notes, and 24- vs 48-hour windows with notes, tabular data, or both for discharge summaries. These manipulations are what let the paper attribute differences in performance to temporal reasoning rather than to dataset or modality alone.","core_discovery":"The central finding is that the bottleneck is temporal reasoning, not context length. On discharge summarization, feeding the model only the last 24 or 48 hours produced modest ROUGE-L scores, with Qwen and RAG reaching the highest value; for daily Assessment and Plan generation, adding one prior day helped, but adding all prior days did not consistently beat a single prior day; and on six EHRShot diagnosis-prediction tasks, models defaulted to the majority class, giving high accuracy but F1 scores near zero for rare diseases such as celiac disease and lupus. RAG improved hallucination in expert review but did not close the gap, and a chain-of-thought event-extraction step did not beat direct generation. The paper argues that these results establish a need for evaluations and models that are temporally grounded, rather than simply long-context or retrieval-heavy.","pith_inferences":["Editorial inference: if the bottleneck is temporal reasoning rather than context length, then a fair test is to construct patient trajectories where the key diagnosis is only inferable from a critical event many days before the summary window; the paper's short-window design leaves that test open.","Editorial inference: the near-zero F1 for rare diseases suggests that future work should report precision and recall rather than accuracy, and could explore counterfactual data augmentation or retrieval of similar rare cases.","Editorial inference: the finding that shuffled tabular data sometimes outperforms chronological tabular data in the 24-hour window hints that near-discharge data is nearly order-independent; outside that window, temporal order may matter more, which is a testable extension.","Editorial inference: since RAG improved hallucination in expert review but not automated metrics, evaluation of longitudinal summarization may need richer clinician-annotated criteria, such as retention of outdated diagnoses, rather than ROUGE and BERTScore alone."],"forward_implications":["Evaluation of clinical summarization should be redesigned so that correct answers require information from several days earlier, rather than a short pre-discharge window.","Retrieval augmentation is a partial mitigation for hallucination but not a substitute for temporal grounding; models need training or prompting that explicitly links events across timestamps.","Rare-disease prediction will need methods beyond majority-class calibration, because raw accuracy is inflated by class imbalance.","Chain-of-thought event extraction offers interpretability but, as currently pipelined, does not improve generation quality; any future use should include adaptation or integration with instruction tuning.","Adding prior context improves progress-note generation up to a point; single-day context can outperform multi-day context in later hospital days, implying diminishing returns from longer history."],"supporting_citations":[{"why":"Supplies the MIMIC-III dataset used for the discharge summarization and Assessment-and-Plan tasks.","marker":"Johnson et al., 2020"},{"why":"Supplies the EHRShot structured longitudinal dataset used for diagnosis prediction.","marker":"Wornow et al., 2023"},{"why":"Defines the Discharge Me task whose three discharge sections the paper re-engineers into a long-context task.","marker":"Xu et al., 2024a"},{"why":"Provides the ProbSum problem-list summarization setup that the Assessment and Plan generation extends.","marker":"Gao et al., 2023a"},{"why":"Documents the lost-in-the-middle effect that motivates RAG as the comparison approach.","marker":"Liu et al., 2024"},{"why":"Motivates the choice of BGE embeddings for the RAG retrieval component.","marker":"Myers et al., 2024"},{"why":"Provides the chain-of-thought prompting method used for the event-extraction step.","marker":"Wei et al., 2022"},{"why":"Supports converting structured tabular EHR values into natural-language text.","marker":"Gao et al., 2024"}],"fun_headline_variants":["Temporal reasoning trips up clinical LLMs","LLMs miss rare diagnoses despite long context","RAG can't fix LLM temporal blind spots","Clinical LLMs: context length isn't the cure","Why LLMs still fail at longitudinal medicine"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The discharge-summarization task assumes that the last 24 or 48 hours of a stay contain all the information needed to write the full discharge summary, so clinically important events from earlier in a long admission are never shown to the model.","fun_headline_variants_meta":{"raw":{"variants":["Temporal reasoning trips up clinical LLMs","LLMs miss rare diagnoses despite long context","RAG can't fix LLM temporal blind spots","Clinical LLMs: context length isn't the cure","Why LLMs still fail at longitudinal medicine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1209,"prompt_tokens":888,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":251}},"tokens_in":504,"tokens_out":321,"duration_ms":3705,"temperature":1.0,"reasoning_tokens":251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:42:17.054115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a long-context LLM a full multi-day admission instead of the last 24 or 48 hours and compare its discharge summary against the same model restricted to the short window; if full-input summaries reach clinician-level completeness and accuracy, the paper's claim that models cannot reason over long longitudinal trajectories would be undermined. A simpler version: find one 72-hour-plus admission where an event on day 1 determines the discharge diagnosis, and check whether the model using only the final window misses it while the full-input model catches it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MIMIC-III dataset used for the discharge summarization and Assessment-and-Plan tasks."}],"review_version":1}