{"id":"a30f0485-ac2a-44a7-a30a-d7a8e659de8b","arxiv_id":"2412.01331","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A code-agnostic text representation of EHRs outperformed code-based input for predicting retinopathy, nephropathy and neuropathy at 1, 5, and 10 years in 133,784 UK type 2 diabetes patients.","lead":"Researchers tested whether turning electronic health records into plain text, instead of clinical codes, improves prediction of long-term diabetes complications. The text-based model beat the code-based model on UK primary care data, and longer context windows helped most.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Negative labels may be unverifiable: for patients without complications the prediction window starts at their last recorded EHR event, so 'no complication' is conflated with 'no follow-up'; this can drive the longer-window result.","rationale":"The reader's weakest assumption identifies the same mechanism, and I agree it is the load-bearing issue. I considered the truncation-direction confound between GatorTron (left) and Biomedical-longformer (right) in Table 5; that is real but secondary, because the label asymmetry alone can produce the paper's headline window effect and can distort any comparison between text and code representations trained on those labels. The strongest claim bundles three assertions: code-agnostic > code-based; longer windows better; context length important. The second assertion is directly downstream of the label construction, and the first and third are evaluated on labels generated by that construction, so all three are exposed. The paper is otherwise internally consistent and includes useful ablations (bootstrap CIs, left-truncation experiments, comparison across pretrained models), and there is no sign of circular reasoning or fabricated results. The issue is addressable by reanalysis rather than by rejecting the approach, so I would keep the reader's CONDITIONAL verdict and require the landmark/censoring analysis as a condition of acceptance.","tokens_in":8709,"tokens_out":8953,"duration_ms":80141,"concrete_test":"Re-run the main experiments with a landmark design: fix the index date for all patients (e.g., first T2DM diagnosis or 12 months after registration), keep only patients with at least 10 years of follow-up or censor at the earliest of deregistration, death, and data cutoff, and label complications occurring before censorship. Recompute Table 2 and the left-truncation table. If text still beats code and the 5- and 10-year AUPRC still exceed the 1-year values, the concern is resolved; if the window trend flattens or reverses, the reported advantage is an artifact of label construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines study entry asymmetrically: for patients with a complication the input ends at the visit before the first complication, while for patients without complications it ends at the last recorded event. For the latter group the prediction window therefore begins exactly at the end of observable data: any complication occurring after transfer, death, or data cutoff is unrecorded, and the patient is labeled negative even if the event occurred. The 1-, 5-, and 10-year labels for this group are all identical (negative), while for the complication group longer windows can only add positive labels because later-occurring complications become visible. This is not merely censoring; it constructs the negative class from zero-follow-up patients and the positive class from patients with observed follow-up. The paper acknowledges the mechanism in Section 4 ('longer prediction windows increases the likelihood of observing a complication'), which is a statement about label availability, not about predictive signal. Since A.1 states patients are linked to ONS death data, censoring at death was available but is not used. Without a landmark design or a minimum follow-up requirement, the longer-window improvement and the absolute micro-F1/AUPRC values in Tables 2 and 4 cannot be interpreted as predictive performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a code-agnostic approach to EHR-based prediction of microvascular complications in type 2 diabetes, replacing clinical codes with their textual descriptors and fine-tuning a pretrained clinical language model (GatorTron) for multi-label prediction of retinopathy, nephropathy, and neuropathy over 1-, 5-, and 10-year windows. Using CPRD AURUM data from 133,784 patients, the authors compare text-based and code-based models, evaluate several pretrained models, and investigate the effect of input truncation and context length. They report that text-based models outperform code-based models, that longer prediction windows yield better micro-F1 and micro-AUPRC, and that left-truncated (recent-first) inputs improve performance. Results include bootstrap confidence intervals and the code is publicly available.","tokens_in":8957,"tokens_out":5374,"duration_ms":47802,"significance":"If the findings hold, the paper makes a useful contribution to ontology-agnostic EHR modeling by demonstrating that textual descriptors of clinical codes can match or exceed code-based representations for a clinically important prediction task, and by showing that context length is a key determinant of performance. The study is built on a large real-world cohort, includes multiple model comparisons, reports bootstrap confidence intervals, and releases code, which are all strengths. However, the central claim that longer prediction windows improve performance is confounded by the label-construction design, which systematically ties observable follow-up to the outcome label. Because this issue directly affects the interpretation of Tables 2 and 4 and one of the abstract's main claims, the current version cannot be accepted without re-analysis or a clear demonstration that the effect survives an appropriate censoring policy.","major_comments":[{"comment":"The label construction is asymmetric and confounds the longer-window analysis. For patients with a complication, study entry is defined as the first event up to the visit before the first recorded complication, so the prediction window is followed by observed data in which a later complication can appear. For patients without a complication, the sequence ends at the last recorded event, so the prediction window starts exactly at the end of observable data: any complication occurring after transfer, death, or data cutoff is unrecorded, and the patient is labeled negative even if the event occurred. The 1-, 5-, and 10-year labels for this group are therefore identical (negative), while for the complication group longer windows can only add positive labels because later-occurring complications become visible. This means the better performance reported for longer windows in Table 2 may reflect label availability rather than predictive signal. Since Appendix A.1 states that patients were linked to ONS death data, censoring at death was available but not used. The authors should adopt a landmark design with a minimum follow-up requirement, or explicitly censor at the last observation date and at death, and then report whether the longer-window improvement persists.","section":"Section 3.1"},{"comment":"The claim that 'models perform better over longer prediction timeframes' is supported only by aggregate metrics whose denominator is changed by the label-construction issue. The paper itself explains the improvement by saying 'the longer prediction windows increases the likelihood of observing a complication,' which is a statement about label availability, not about predictive performance. Under the current design, the positive prevalence mechanically increases with the window length for the complication group while the negative group is fixed, so micro-F1 and micro-AUPRC can improve even if the model has no additional predictive skill at longer horizons. The authors should either re-analyze with a time-to-event formulation (e.g., censored survival or incident dynamic AUC) or restrict to patients with complete follow-up for each window. Without such an analysis, the abstract statement about longer windows being better is not supported by the evidence as presented.","section":"Section 4, Table 2"},{"comment":"The description of the fine-tuning procedure is ambiguous and potentially inconsistent with the claim of held-out evaluation. The text states: 'We split our data 80/10/10 into training, test and validation' and then 'fine-tuned on the entire dataset for 48000 steps with early stopping.' If 'entire dataset' is meant literally, the test set would be used during training, invalidating all reported results. Please clarify that fine-tuning used only the training split. Additionally, Appendix A.2 says the learning rate was searched to give the 'lowest F1 score,' but lower F1 is worse, not better; this is presumably a typo for either the lowest validation loss or the highest validation F1. The current wording makes the hyperparameter selection procedure impossible to reproduce and raises doubts about whether the reported test-set numbers come from a properly held-out protocol.","section":"Section 3.3 and Appendix A.2"}],"minor_comments":[{"comment":"Per-class F1 and recall are reported without confidence intervals or significance tests. The claim that retinopathy is the highest-performing class would be strengthened by quantifying the uncertainty, especially because class frequencies are highly imbalanced (Table 1).","section":"Tables 3 and 4"},{"comment":"The statement that text-based models outperformed code-based models 'although not at all time windows' is vague; please specify which windows and which metrics show statistically significant differences after the Bonferroni correction.","section":"Section 4, first paragraph"},{"comment":"It is not explicitly stated whether Table 2 uses right-truncation and Table 4 uses left-truncation. Since the comparison between the tables is central to the truncation claim, state the truncation direction for each table explicitly.","section":"Section 4, 'Restrictions on context length'"},{"comment":"When concatenating textual descriptors chronologically, clarify what delimiter separates events and whether the descriptor for each code is taken verbatim or normalized (e.g., lowercased, punctuation removed). This affects tokenization and reproducibility.","section":"Section 3.2"},{"comment":"The phrase 'searched for a learning rate that gave the lowest F1 score' appears to be a typo; lower F1 is worse. Please correct to 'lowest validation loss' or 'highest validation F1.'","section":"Appendix A.2"},{"comment":"The phenotype definitions are attributed to a GitHub repository (Eto, 2023) without a version or persistent identifier. Please cite the specific release or a peer-reviewed description of the phenotype definitions.","section":"Appendix A.1"},{"comment":"The input-format figure would be more informative with a short concrete example showing a few events in both text and code form, rather than a schematic.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The label-construction problem in Section 3.1 is the main substantive concern and it is acknowledged in the paper's own explanation in Section 4. The authors will need to demonstrate that the longer-window result is not an artifact of follow-up time. The code-agnostic versus code-based comparison is likely more robust and could survive the re-analysis. The 'fine-tuned on the entire dataset' wording is worrying but probably a typo; still, it must be corrected. The paper is within scope for ML4H Findings and the empirical setup is otherwise solid, so a major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid empirical comparison of text-based vs code-based EHR representations on T2DM microvascular complication prediction, using a large UK cohort. It does several things well: the text-descriptor approach is a sensible extension of prior work, the multi-label 1/5/10-year setup is clearly described, the bootstrap CIs and pairwise testing are appropriate, and the context-length analysis (left vs right truncation, Longformer comparison) is genuinely informative. The authors are also honest about the mechanism behind their longer-window result: they attribute it to 'increasing the likelihood of observing a complication,' which is exactly the problem.\n\nThat sentence is the core weakness. Study entry for patients without complications is defined as the last recorded EHR event, so their prediction window starts where observability ends. For those patients, a negative label at 1, 5, and 10 years is not a claim about not having the complication; it is a claim about not observing it before transfer, death, or data cutoff. The paper even has ONS death linkage available but does not use it. This is not a minor nuisance. It means the 'longer windows perform better' result is largely an artifact of label availability, and the absolute AUPRC/F1 numbers in Tables 2 and 4 cannot be interpreted as predictive performance. This is addressable with a landmark design, minimum follow-up, or censoring-aware labels, but it has to be fixed.\n\nSmaller issues: per-class F1/recall in Tables 3-4 have no uncertainty estimates, and the fine-tuning description in A.2 is confusing ('lowest F1 score' during learning-rate search; probably a typo). These are minor by comparison.\n\nThe text-vs-code comparison itself is internally consistent and the direction of the finding (text helps) is plausible, but the magnitude of the advantage is confounded by the same label construction, since both models use the same labels. So the central claim that a code-agnostic approach 'outperforms' is weakened too, though less severely.\n\nWho gets value from this? Anyone working on ontology-agnostic EHR modeling will want to know this analysis exists and where it falls short. It deserves a serious referee, but a referee should push hard on the design before any clinical or practical conclusion is drawn. I would send it to review with the expectation of major revision rather than desk reject. For my own work, I would not cite the quantitative results in their current form.\n\nNet: worth engaging, but the label construction is load-bearing and needs to be redone or explicitly acknowledged as a limitation with sensitivity analyses.","headline":"Useful text-vs-code EHR comparison, but the label construction conflates 'no complication' with 'no follow-up', so the longer-window claim needs major revision before the results can be taken at face value.","tokens_in":9496,"tokens_out":1106,"would_cite":false,"duration_ms":11866,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-based EHR models beat code-based ones for diabetes risk.","keywords":["code-agnostic EHR models","clinical language models","multi-label classification","type 2 diabetes","microvascular complications","long-term prediction","pretrained transformers","EHR text representation"],"falsifier":"Restrict the evaluation to patients with at least 10 years of continuous registration after study entry and recompute the 5- and 10-year micro-averaged AUPRC for text versus code models; if the text advantage and the longer-window improvement shrink or vanish, the performance gains are driven by label construction rather than by the representation.","tokens_in":8530,"feed_emoji":"🩺","tokens_out":8196,"duration_ms":66398,"temperature":0.7,"pith_summary":"This paper tests whether an electronic health record (EHR) model can predict long-term microvascular complications of Type 2 diabetes without depending on any particular clinical coding system. The authors replace every clinical code in a patient's record with its natural-language description, concatenate these descriptions chronologically, and fine-tune a pretrained clinical language model to predict retinopathy, nephropathy, and neuropathy simultaneously at 1-, 5-, and 10-year horizons. On 133,784 UK primary-care records they report that this code-agnostic text approach outperforms an equivalent model trained on raw clinical codes, that performance is generally better at longer prediction windows, and that giving the model more recent context or a longer context window helps most. If the finding holds, it means EHR predictive models could be moved across hospitals and registries without costly ontology mapping, and that pretrained language models already carry useful clinical knowledge.","feed_headline":"Text beats codes for predicting diabetes complications","feed_subtitle":"A code-agnostic model reads plain-text clinical descriptors and predicts microvascular complications up to 10 years out","key_machinery":"The central object is the code-agnostic EHR representation: each structured clinical code is replaced by its associated textual descriptor, and all descriptors in a patient's record are concatenated in chronological order to form a text sequence. This sequence is tokenized, truncated or padded to the model's context limit, and passed through a fine-tuned pretrained clinical transformer with a linear output head that applies sigmoid activations to produce independent probabilities for retinopathy, nephropathy, and neuropathy. The text representation does the load-bearing work: it lets the model draw on knowledge already embedded in pretrained language models, removes dependence on any single coding ontology, and, because descriptors are shared across registries, opens the door to combining datasets without expert mapping. The authors also identify context length as a mechanism: with a median EHR length of 2,272 tokens, the 512-token model truncates most records, and preserving recent events or using a longer context window both improve performance.","core_discovery":"The central claim is that representing a patient's EHR as a sequence of textual descriptors rather than as a sequence of clinical codes yields better predictions of first microvascular complications in Type 2 diabetes. Using GatorTron-base fine-tuned for each of the three windows, the authors report micro-averaged area under the precision-recall curve of 0.44, 0.51, and 0.50 for the text-based model at 1, 5, and 10 years versus 0.40, 0.43, and 0.47 for the code-based model. They further show that truncating from the left, keeping the most recent events in the way clinicians read records, raises the metric to 0.64, 0.66, and 0.69, and that a model with a 4096-token context window reaches 0.57, 0.63, and 0.64. The paper also reports that gains are concentrated in the most common first complication, retinopathy, meaning the multi-label setup is biased toward the first occurring complication. The authors conclude that code-agnostic text representations are a viable path to generalisable clinical prediction models, with context length a key determinant of performance.","pith_inferences":["A natural next step the authors do not take is to pool records from different countries whose coding systems differ; the code-agnostic representation would make such pooling direct, provided the textual descriptors are kept in a common language.","Because left truncation worked best, a sliding-window design that always feeds a fixed number of the most recent events would be a cheap way to scale the approach to much longer patient histories.","Another testable consequence is that translating a new ontology's codes into English descriptors should transfer without retraining, a claim that could be checked by holding out one registry entirely."],"forward_implications":["If text descriptors beat raw codes, EHR prediction models can be reused across primary, secondary, and national registries without translating between different code systems.","Because left truncation improves every window, models should weight recent history more heavily rather than treating all history equally.","Longer context windows improve performance enough that context length, not model architecture, is a primary limit on current EHR language models.","The multi-label model's bias toward retinopathy, the most frequent first complication, means that predictions for later-occurring complications need either different supervision or models that see past the first complication.","Even a general-purpose model not trained on clinical text performs close to the clinical model on this task, suggesting the gain comes largely from the text representation itself rather than from clinical pretraining."],"supporting_citations":[{"why":"Introduced the code-agnostic idea of feeding textual descriptors of clinical codes into a pretrained encoder and reported gains over code embeddings; this paper extends that setup to long-term multi-label T2DM prediction.","marker":"Munoz-Farre et al. (2022)"},{"why":"Showed that text-based code embeddings with BERT perform comparably across model designs, motivating the comparison of pretrained models here.","marker":"Hur et al. (2022)"},{"why":"Provided GatorTron-base, the pretrained clinical language model used as the main encoder for the text and code sequences.","marker":"Yang et al. (2022)"},{"why":"Supplied the Longformer architecture that gives Biomedical-longformer its 4096-token context, used to test the effect of longer input.","marker":"Beltagy et al. (2020)"},{"why":"Describes the UK primary-care EHR dataset from which the 133,784-patient cohort is drawn.","marker":"Wolf et al. (2019)"},{"why":"Provided the validated phenotype definitions used to identify T2DM and microvascular complications and first diagnosis dates.","marker":"Eto (2023)"},{"why":"Provided BERT-base, used as a non-clinical pretrained baseline to isolate the value of clinical pretraining.","marker":"Devlin et al. (2019)"}],"fun_headline_variants":["Text beats codes for predicting diabetes complications up to 10 years","Plain-text EHR model outperforms code-based for diabetes risks","Clinical text predicts diabetes microvascular events better than codes","Diabetes complication prediction: text wins over codes at long horizons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that a patient with no recorded complication before the end of their EHR history truly did not develop one; no accounting is made for patients who left the practice, died, or were otherwise lost to follow-up before the prediction window elapsed.","fun_headline_variants_meta":{"raw":{"variants":["Text beats codes for predicting diabetes complications up to 10 years","Plain-text EHR model outperforms code-based for diabetes risks","Clinical text predicts diabetes microvascular events better than codes","Diabetes complication prediction: text wins over codes at long horizons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1613,"prompt_tokens":1011,"completion_tokens":602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":535}},"tokens_in":627,"tokens_out":602,"duration_ms":6039,"temperature":1.0,"reasoning_tokens":535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:27:56.323384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Restrict the evaluation to patients with at least 10 years of continuous registration after study entry and recompute the 5- and 10-year micro-averaged AUPRC for text versus code models; if the text advantage and the longer-window improvement shrink or vanish, the performance gains are driven by label construction rather than by the representation.","supporting_citations":[{"cited_title":"sEHR-CE: Language modelling of structured EHR data for efficient and generalizable patient cohort expansion","cited_arxiv_id":"2211.17121","evidence_quote":"Introduced the code-agnostic idea of feeding textual descriptors of clinical codes into a pretrained encoder and reported gains over code embeddings; this paper extends that setup to long-term multi-label T2DM prediction."},{"cited_title":"Unifying Heterogeneous Electronic Health Records Systems via Text - Based Code Embedding","cited_arxiv_id":null,"evidence_quote":"Showed that text-based code embeddings with BERT perform comparably across model designs, motivating the comparison of pretrained models here."},{"cited_title":"Data resource profile: Clinical Practice Research Datalink ( CPRD ) Aurum","cited_arxiv_id":null,"evidence_quote":"Describes the UK primary-care EHR dataset from which the 133,784-patient cohort is drawn."},{"cited_title":"MULTIPLY - Initiative , August 2023","cited_arxiv_id":null,"evidence_quote":"Provided the validated phenotype definitions used to identify T2DM and microvascular complications and first diagnosis dates."}],"review_version":1}