Pith. sign in

REVIEW 3 major objections 7 minor 19 references

Exploring Long-Term Prediction of Type 2 Diabetes Microvascular Complications

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Text-based EHR models beat code-based ones for diabetes risk.

desk verdict Useful text-vs-code EHR comparison, but the label construction conflates 'no complication' with 'no follow-up', so the longer-window claim needs major revision before the results can be taken at face value. read the letter →

arxiv 2412.01331 v1 pith:2AARMRGW submitted 2024-12-02 cs.LG cs.CL

classification cs.LGcs.CL
keywords code-agnosticEHRmodelsclinicallanguagemulti-labelclassificationtype2diabetesmicrovascularcomplicationslong-termpredictionpretrainedtransformerstextrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether an electronic health record (EHR) model can predict long-term microvascular complications of Type 2 diabetes without depending on any particular clinical coding system. The authors replace every clinical code in a patient's record with its natural-language description, concatenate these descriptions chronologically, and fine-tune a pretrained clinical language model to predict retinopathy, nephropathy, and neuropathy simultaneously at 1-, 5-, and 10-year horizons. On 133,784 UK primary-care records they report that this code-agnostic text approach outperforms an equivalent model trained on raw clinical codes, that performance is generally better at longer prediction windows, and that giving the model more recent context or a longer context window helps most. If the finding holds, it means EHR predictive models could be moved across hospitals and registries without costly ontology mapping, and that pretrained language models already carry useful clinical knowledge.

What carries the argument

The central object is the code-agnostic EHR representation: each structured clinical code is replaced by its associated textual descriptor, and all descriptors in a patient's record are concatenated in chronological order to form a text sequence. This sequence is tokenized, truncated or padded to the model's context limit, and passed through a fine-tuned pretrained clinical transformer with a linear output head that applies sigmoid activations to produce independent probabilities for retinopathy, nephropathy, and neuropathy. The text representation does the load-bearing work: it lets the model draw on knowledge already embedded in pretrained language models, removes dependence on any single coding ontology, and, because descriptors are shared across registries, opens the door to combining datasets without expert mapping. The authors also identify context length as a mechanism: with a median EHR length of 2,272 tokens, the 512-token model truncates most records, and preserving recent events or using a longer context window both improve performance.

What would settle it

Restrict the evaluation to patients with at least 10 years of continuous registration after study entry and recompute the 5- and 10-year micro-averaged AUPRC for text versus code models; if the text advantage and the longer-window improvement shrink or vanish, the performance gains are driven by label construction rather than by the representation.

Watch

Extended reading notes

Core claim

The central claim is that representing a patient's EHR as a sequence of textual descriptors rather than as a sequence of clinical codes yields better predictions of first microvascular complications in Type 2 diabetes. Using GatorTron-base fine-tuned for each of the three windows, the authors report micro-averaged area under the precision-recall curve of 0.44, 0.51, and 0.50 for the text-based model at 1, 5, and 10 years versus 0.40, 0.43, and 0.47 for the code-based model. They further show that truncating from the left, keeping the most recent events in the way clinicians read records, raises the metric to 0.64, 0.66, and 0.69, and that a model with a 4096-token context window reaches 0.57, 0.63, and 0.64. The paper also reports that gains are concentrated in the most common first complication, retinopathy, meaning the multi-label setup is biased toward the first occurring complication. The authors conclude that code-agnostic text representations are a viable path to generalisable clinical prediction models, with context length a key determinant of performance.

Load-bearing premise

The results assume that a patient with no recorded complication before the end of their EHR history truly did not develop one; no accounting is made for patients who left the practice, died, or were otherwise lost to follow-up before the prediction window elapsed.

Editorial extensions

If this is right

  • If text descriptors beat raw codes, EHR prediction models can be reused across primary, secondary, and national registries without translating between different code systems.
  • Because left truncation improves every window, models should weight recent history more heavily rather than treating all history equally.
  • Longer context windows improve performance enough that context length, not model architecture, is a primary limit on current EHR language models.
  • The multi-label model's bias toward retinopathy, the most frequent first complication, means that predictions for later-occurring complications need either different supervision or models that see past the first complication.
  • Even a general-purpose model not trained on clinical text performs close to the clinical model on this task, suggesting the gain comes largely from the text representation itself rather than from clinical pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the authors do not take is to pool records from different countries whose coding systems differ; the code-agnostic representation would make such pooling direct, provided the textual descriptors are kept in a common language.
  • Because left truncation worked best, a sliding-window design that always feeds a fixed number of the most recent events would be a cheap way to scale the approach to much longer patient histories.
  • Another testable consequence is that translating a new ontology's codes into English descriptors should transfer without retraining, a claim that could be checked by holding out one registry entirely.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a code-agnostic approach to EHR-based prediction of microvascular complications in type 2 diabetes, replacing clinical codes with their textual descriptors and fine-tuning a pretrained clinical language model (GatorTron) for multi-label prediction of retinopathy, nephropathy, and neuropathy over 1-, 5-, and 10-year windows. Using CPRD AURUM data from 133,784 patients, the authors compare text-based and code-based models, evaluate several pretrained models, and investigate the effect of input truncation and context length. They report that text-based models outperform code-based models, that longer prediction windows yield better micro-F1 and micro-AUPRC, and that left-truncated (recent-first) inputs improve performance. Results include bootstrap confidence intervals and the code is publicly available.

Significance. If the findings hold, the paper makes a useful contribution to ontology-agnostic EHR modeling by demonstrating that textual descriptors of clinical codes can match or exceed code-based representations for a clinically important prediction task, and by showing that context length is a key determinant of performance. The study is built on a large real-world cohort, includes multiple model comparisons, reports bootstrap confidence intervals, and releases code, which are all strengths. However, the central claim that longer prediction windows improve performance is confounded by the label-construction design, which systematically ties observable follow-up to the outcome label. Because this issue directly affects the interpretation of Tables 2 and 4 and one of the abstract's main claims, the current version cannot be accepted without re-analysis or a clear demonstration that the effect survives an appropriate censoring policy.

major comments (3)
  1. [Section 3.1] The label construction is asymmetric and confounds the longer-window analysis. For patients with a complication, study entry is defined as the first event up to the visit before the first recorded complication, so the prediction window is followed by observed data in which a later complication can appear. For patients without a complication, the sequence ends at the last recorded event, so the prediction window starts exactly at the end of observable data: any complication occurring after transfer, death, or data cutoff is unrecorded, and the patient is labeled negative even if the event occurred. The 1-, 5-, and 10-year labels for this group are therefore identical (negative), while for the complication group longer windows can only add positive labels because later-occurring complications become visible. This means the better performance reported for longer windows in Table 2 may reflect label availability rather than predictive signal. Since Appendix A.1 states that patients were linked to ONS death data, censoring at death was available but not used. The authors should adopt a landmark design with a minimum follow-up requirement, or explicitly censor at the last observation date and at death, and then report whether the longer-window improvement persists.
  2. [Section 4, Table 2] The claim that 'models perform better over longer prediction timeframes' is supported only by aggregate metrics whose denominator is changed by the label-construction issue. The paper itself explains the improvement by saying 'the longer prediction windows increases the likelihood of observing a complication,' which is a statement about label availability, not about predictive performance. Under the current design, the positive prevalence mechanically increases with the window length for the complication group while the negative group is fixed, so micro-F1 and micro-AUPRC can improve even if the model has no additional predictive skill at longer horizons. The authors should either re-analyze with a time-to-event formulation (e.g., censored survival or incident dynamic AUC) or restrict to patients with complete follow-up for each window. Without such an analysis, the abstract statement about longer windows being better is not supported by the evidence as presented.
  3. [Section 3.3 and Appendix A.2] The description of the fine-tuning procedure is ambiguous and potentially inconsistent with the claim of held-out evaluation. The text states: 'We split our data 80/10/10 into training, test and validation' and then 'fine-tuned on the entire dataset for 48000 steps with early stopping.' If 'entire dataset' is meant literally, the test set would be used during training, invalidating all reported results. Please clarify that fine-tuning used only the training split. Additionally, Appendix A.2 says the learning rate was searched to give the 'lowest F1 score,' but lower F1 is worse, not better; this is presumably a typo for either the lowest validation loss or the highest validation F1. The current wording makes the hyperparameter selection procedure impossible to reproduce and raises doubts about whether the reported test-set numbers come from a properly held-out protocol.
minor comments (7)
  1. [Tables 3 and 4] Per-class F1 and recall are reported without confidence intervals or significance tests. The claim that retinopathy is the highest-performing class would be strengthened by quantifying the uncertainty, especially because class frequencies are highly imbalanced (Table 1).
  2. [Section 4, first paragraph] The statement that text-based models outperformed code-based models 'although not at all time windows' is vague; please specify which windows and which metrics show statistically significant differences after the Bonferroni correction.
  3. [Section 4, 'Restrictions on context length'] It is not explicitly stated whether Table 2 uses right-truncation and Table 4 uses left-truncation. Since the comparison between the tables is central to the truncation claim, state the truncation direction for each table explicitly.
  4. [Section 3.2] When concatenating textual descriptors chronologically, clarify what delimiter separates events and whether the descriptor for each code is taken verbatim or normalized (e.g., lowercased, punctuation removed). This affects tokenization and reproducibility.
  5. [Appendix A.2] The phrase 'searched for a learning rate that gave the lowest F1 score' appears to be a typo; lower F1 is worse. Please correct to 'lowest validation loss' or 'highest validation F1.'
  6. [Appendix A.1] The phenotype definitions are attributed to a GitHub repository (Eto, 2023) without a version or persistent identifier. Please cite the specific release or a peer-reviewed description of the phenotype definitions.
  7. [Figure 1] The input-format figure would be more informative with a short concrete example showing a few events in both text and code form, rather than a schematic.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical benchmark on held-out labels, not a derivation that reduces to its own fitted inputs.

full rationale

The central claims are that a text-descriptor representation outperforms a code-based representation, that longer prediction windows and longer context lengths improve performance, and that the multi-label design is biased toward the first-occurring complication. These are empirical model comparisons, not derivation steps that re-import their own outputs. Section 3.3 states: 'We split our data 80/10/10 into training, test and validation using stratified sampling to ensure the imbalance remained the same... All results are presented calculated on the held out test set.' No fitted parameter is renamed as a prediction, and no outcome label is used as an input. The text and code representations are alternative encodings of the same EHR events (Section 3.2), and both are evaluated against the same held-out complication labels, so the code-agnostic advantage is not true by construction. The longer-window result is explicitly attributed by the authors to label availability: 'the longer prediction windows increases the likelihood of observing a complication' (Section 4). That is a censoring and outcome-construction concern about what the metric reflects, not circular reasoning: the labels are targets, not predictors, and no equation defines the prediction in terms of the target. There are no load-bearing self-citations (prior work cited, Munoz-Farre et al. 2022 and Hur et al. 2022, is external), no uniqueness theorem imported from the authors' own work, and no ansatz smuggled in via citation. The architecture is a standard fine-tuned clinical language model with a linear output layer. Any weaknesses in negative-label construction, follow-up duration, or censoring are correctness or validity risks and should be scored as such, not as circularity. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result rests on cohort and label definitions that are reasonable but unverified, especially the assumption that an unrecorded complication is a non-event. The model itself introduces no invented entities and few free parameters beyond standard hyperparameters and context-length choices.

free parameters (3)
  • learning rate = grid {1e-3, 2e-5, 3e-5, 4e-5, 5e-5}, chosen by F1; final value not reported
    Appendix A.2 states each model was searched for the learning rate giving the 'lowest F1 score' (likely a typo for highest), then fine-tuned; the chosen value affects all reported results.
  • class weights for weighted cross entropy = not reported
    Weighted cross entropy is used to counter label imbalance, but the exact weights are not specified; they are derived from class frequencies and affect performance.
  • context length and truncation side = 512 tokens left-truncated for GatorTron/BERT; 4096 right-truncated default for Biomedical-longformer
    Context length and truncation direction are modeling choices that directly drive the reported gains; left truncation improved results, and the long context model performed best.
assumptions (4)
  • domain assumption The first recorded diagnosis date for a complication correctly identifies true onset, and complications occurring before T2DM diagnosis are correctly excluded.
    Section 3.1 relies on validated phenotype definitions from the MULTIPLY repository (Eto, 2023), not adjudicated chart review, to define outcomes.
  • domain assumption Patients with no recorded complication up to their last EHR event did not develop the complication during the prediction window.
    Section 3.1 defines negative labels this way without accounting for follow-up time or censoring; this is the weakest assumption and affects the longer-window claim.
  • domain assumption Textual descriptors of clinical codes retain enough clinical information for prediction and are equivalent across registries.
    This is the core premise of the code-agnostic approach; the paper shows moderate performance but does not test on another ontology or registry.
  • domain assumption Eligibility criteria (age >= 18, permanent registration, HES/ONS linkage, at least 3 unique events, records 1985-2020) do not bias the cohort in a way that changes the conclusions.
    Appendix A.1 restricts the cohort to linked London patients; this may limit generalizability to other regions or populations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Long-Term Prediction of Type 2 Diabetes Microvascular Complications." pith.science (2026). https://pith.science/paper/2AARMRGW

@misc{pith2026241201331,
  author       = {Pith},
  title        = {Pith review of: Exploring Long-Term Prediction of Type 2 Diabetes Microvascular Complications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AARMRGW}},
  note         = {Machine review of arXiv:2412.01331}
}
read the original abstract

Electronic healthcare records (EHR) contain a huge wealth of data that can support the prediction of clinical outcomes. EHR data is often stored and analysed using clinical codes (ICD10, SNOMED), however these can differ across registries and healthcare providers. Integrating data across systems involves mapping between different clinical ontologies requiring domain expertise, and at times resulting in data loss. To overcome this, code-agnostic models have been proposed. We assess the effectiveness of a code-agnostic representation approach on the task of long-term microvascular complication prediction for individuals living with Type 2 Diabetes. Our method encodes individual EHRs as text using fine-tuned, pretrained clinical language models. Leveraging large-scale EHR data from the UK, we employ a multi-label approach to simultaneously predict the risk of microvascular complications across 1-, 5-, and 10-year windows. We demonstrate that a code-agnostic approach outperforms a code-based model and illustrate that performance is better with longer prediction windows but is biased to the first occurring complication. Overall, we highlight that context length is vitally important for model performance. This study highlights the possibility of including data from across different clinical ontologies and is a starting point for generalisable clinical models.

Figures

Figures reproduced from arXiv: 2412.01331 by the authors.

Figure 1
Figure 1. Input Format for Text and Code-based Approaches [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Multi-label Approach and Model Architec￾ture 7 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. that many EHRs are still truncated when using Biomedical-longformer as they fall over 4096 tokens [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

  1. [1]

    Incidence rates and predictors of microvascular and macrovascular complications in patients with type 2 diabetes: Results from the longitudinal global discover study

    Suzanne V Arnold, Kamlesh Khunti, Fengming Tang, Hungta Chen, Javier Cid-Ruzafa, Andrew Cooper, Peter Fenici, Marilia B Gomes, Niklas Hammar, Linong Ji, Gabriela Luporini Saraiva, Jesús Medina, Antonio Nicolucci, Larisa Ramirez, Wolfgang Rathmann, Marina V Shestakova, Iichiro Shimomura, Filip Surmont, Jiten Vora, Hirotaka Watada, and Mikhail Kosiborod. In...

  2. [2]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long - Document Transformer , December 2020. URL http://arxiv.org/abs/2004.05150. arXiv:2004.05150 [cs]

  3. [3]

    Jack R. W. Brownrigg, Cian O. Hughes, David Burleigh, Alan Karthikesalingam, Benjamin O. Patterson, Peter J. Holt, Matthew M. Thompson, Simon de Lusignan, Kausik K. Ray, and Robert J. Hinchliffe. Microvascular disease and risk of cardiovascular events among individuals with type 2 diabetes: a population-level cohort study. The Lancet Diabetes & Endocrinol...

  4. [4]

    Epidemiology of Diabetes and Diabetes - Related Complications

    Anjali D Deshpande, Marcie Harris-Hayes, and Mario Schootman. Epidemiology of Diabetes and Diabetes - Related Complications . Physical Therapy, 88 0 (11): 0 1254--1264, November 2008. ISSN 0031-9023. doi:10.2522/ptj.20080020. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3870323/

  5. [5]

    BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding . In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies ...

  6. [6]

    MULTIPLY - Initiative , August 2023

    Fabiola Eto. MULTIPLY - Initiative , August 2023. URL https://github.com/Fabiola-Eto/MULTIPLY-Initiative. original-date: 2020-11-25T17:37:13Z

  7. [7]

    Untangling the complexity of multimorbidity with machine learning

    Abdelaali Hassaine, Gholamreza Salimi-Khorshidi, Dexter Canoy, and Kazem Rahimi. Untangling the complexity of multimorbidity with machine learning. Mechanisms of Ageing and Development, 190: 0 111325, September 2020. ISSN 0047-6374. doi:10.1016/j.mad.2020.111325. URL https://www.sciencedirect.com/science/article/pii/S0047637420301214

  8. [8]

    Unifying Heterogeneous Electronic Health Records Systems via Text - Based Code Embedding

    Kyunghoon Hur, Jiyoung Lee, Jungwoo Oh, Wesley Price, Younghak Kim, and Edward Choi. Unifying Heterogeneous Electronic Health Records Systems via Text - Based Code Embedding . In Proceedings of the Conference on Health , Inference , and Learning , pages 183--203. PMLR, April 2022. URL https://proceedings.mlr.press/v174/hur22a.html. ISSN: 2640-3498

Show all 19 references
  1. [9]

    Weld, Luke Zettlemoyer, and Omer Levy

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT : Improving Pre -training by Representing and Predicting Spans , January 2020. URL http://arxiv.org/abs/1907.10529. arXiv:1907.10529 [cs]

  2. [10]

    H. Khalil. Diabetes microvascular complications— A clinical update. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 11: 0 S133--S139, November 2017. ISSN 1871-4021. doi:10.1016/j.dsx.2016.12.022. URL https://www.sciencedirect.com/science/article/pii/S1871402116302648

  3. [11]

    Microvascular complications and their associated risk factors in type 2 diabetes mellitus

    Parvin Akter Khanam, Sayama Hoque, Tanjima Begum, Samira Humaira Habib, and Zafar Ahmed Latif. Microvascular complications and their associated risk factors in type 2 diabetes mellitus. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 11: 0 S577--S581, December 2017...

  4. [12]

    Apocrita - High Performance Computing Cluster for Queen Mary University of London

    Thomas King, Simon Butcher, and Lukasz Zalewski. Apocrita - High Performance Computing Cluster for Queen Mary University of London . March 2017. doi:10.5281/zenodo.438045. URL https://zenodo.org/records/438045. Publisher: Zenodo

  5. [13]

    BEHRT : Transformer for Electronic Health Records

    Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi. BEHRT : Transformer for Electronic Health Records . Scientific Reports, 10 0 (1): 0 7155, April 2020. ISSN 2045-23...

  6. [14]

    Hi- BEHRT : Hierarchical Transformer - Based Model for Accurate Prediction of Clinical Events Using Multimodal Longitudinal Electronic Health Records

    Yikuan Li, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewicz, and Kazem Rahimi. Hi- BEHRT : Hierarchical Transformer - Based Model for Accurate Prediction of Clinical Events Using Multimodal Longitudinal Electronic...

  7. [15]

    Vascular complications of diabetes: A narrative review

    Yongxia Lu, Wei Wang, Jingyu Liu, Min Xie, Qiang Liu, and Sufang Li. Vascular complications of diabetes: A narrative review. Medicine, 102 0 (40): 0 e35285, October 2023. ISSN 0025-7974. doi:10.1097/MD.0000000000035285. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10553000/

  8. [16]

    sEHR - CE : Language modelling of structured EHR data for efficient and generalizable patient cohort expansion, November 2022

    Anna Munoz-Farre, Harry Rose, and Sera Aylin Cakiroglu. sEHR - CE : Language modelling of structured EHR data for efficient and generalizable patient cohort expansion, November 2022. URL http://arxiv.org/abs/2211.17121. arXiv:2211.17121 [cs, stat]

  9. [17]

    Data resource profile: Clinical Practice Research Datalink ( CPRD ) Aurum

    Achim Wolf, Daniel Dedman, Jennifer Campbell, Helen Booth, Darren Lunn, Jennifer Chapman, and Puja Myles. Data resource profile: Clinical Practice Research Datalink ( CPRD ) Aurum . International Journal of Epidemiology, 48 0 (6): 0 1740--1740g, December 2019. ISSN 0300-5771, ...

  10. [18]

    Pfeffer, Jason Fries, and Nigam H

    Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A. Pfeffer, Jason Fries, and Nigam H. Shah. The shaky foundations of large language models and foundation models for electronic health records. npj Digital Medicine, 6 0 (1): 0 1--10, J...

  11. [19]

    Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B

    Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B. Costa, Mona G. Flores, Ying Zhang, Tanja Magoc, Christopher A. Harle, Gloria Lipori, Duane A. Mitchell, William R. Hogan, Elizabeth A. Shenkman...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.