Pith. sign in

REVIEW 3 major objections 4 minor 47 references

LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned small NER models, trained with LLMs only as data generators, outscore GPT-4o and prior systems on clinical de-identification in eight languages.

desk verdict English de-identification result is credible, but the multilingual F1 scores are self-evaluations on translated/synthetic data, so the headline 'most accurate' claim overreaches. read the letter →

arxiv 2412.10918 v1 pith:NWBLL35Q submitted 2024-12-14 cs.CL cs.AI

Murat Gunay , Bunyamin Keles , Raife Hizlan This is my paper · ORCID
classification cs.CLcs.AI
keywords de-identificationprotectedhealthinformationnamedentityrecognitionmultilingualNERLLM-in-the-loopi2b2-2014clinicalNLPsyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to prove that de-identifying protected health information does not require a large language model at inference time. The authors argue that small, domain-specific NER models built through an LLM-in-the-loop pipeline, where LLMs only generate and translate training data, can match or beat general-purpose LLMs such as GPT-4o on the English i2b2-2014 benchmark while also covering seven additional languages. If the claim is correct, hospitals could anonymize records locally, keeping sensitive data off external APIs and cutting cost, without sacrificing accuracy. The reported micro-F1 averages are 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 for English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic. A sympathetic reader would take the paper as evidence that specialized small models, not LLMs, are the right tool for this privacy-sensitive task.

What carries the argument

The 'LLM-in-the-loop' development pipeline is the mechanism that carries the argument: LLMs are used to create training data, not to make predictions. English clinical texts are translated into German, Italian, French, Romanian, Turkish, Spanish, and Arabic using the authors' own medical translation models, and LLM-assisted synthetic data generation and labeling expand the non-English corpora. A fake-chunk augmentation step replaces entity spans with realistic alternatives in each language. The final models are fine-tuned small transformers (deberta-v3-small for English; per-language BERT-family models for the others) paired with a rule-based regex layer for structured identifiers like ZIP, PHONE, and SSN. Evaluation uses strict matching, so a prediction counts only when both the text span and the PHI label agree exactly.

What would settle it

Release the models and run them, under strict span-and-label matching, on a held-out set of authentic, manually annotated discharge and admission notes in the seven non-English languages; if macro F1 falls materially below the reported 0.92–0.98 range, the multilingual claim is refuted. A quicker check on the English side is to re-run the i2b2 benchmark with the regex rule layer disabled to see how much of the F1 depends on pattern matching.

Watch

Extended reading notes

Core claim

The central claim is that fine-tuned expert small NER models, developed with the LLM-in-the-loop methodology, are the most accurate healthcare anonymization solutions available. On the i2b2-2014 English test set, the authors' 18-label model reaches a macro F1 of 0.931, beating the best compared prior systems (0.919, 0.863, and 0.840 macro F1) and far exceeding GPT-4o, which scores 0.548 macro and 0.591 micro F1 on the same data. For the non-English languages, the per-label tables report macro-F1 averages of 0.963 for Turkish, 0.960 for German, 0.957 for Spanish, 0.955 for Italian, 0.937 for French, 0.930 for Romanian, and 0.922 for Arabic, with the abstract quoting micro-F1 averages between 0.953 and 0.978. The paper concludes that small models running on-premises can replace API-based LLMs for de-identification and that the pipeline transfers across languages and to other clinical NLP tasks.

Load-bearing premise

The load-bearing premise is that the non-English test sets, built by translating English records and generating synthetic text with LLMs, genuinely represent real clinical notes in those languages; if they do not, the reported near-perfect scores will not carry over to production data.

Editorial extensions

If this is right

  • Hospitals and research groups can de-identify records on-premises with small models, eliminating the privacy and compliance burden of sending PHI to third-party APIs.
  • The reported English benchmark suggests a well-trained small model can outperform a general-purpose LLM on this task, so API-based approaches are not necessary for state-of-the-art de-identification.
  • The LLM-in-the-loop data-generation recipe could be reused for non-English clinical NLP tasks where annotated data are scarce, including the biomedical entity and relation extraction the paper names as future work.
  • Multilingual de-identification at this reported accuracy would make non-English EHRs usable for research, assuming the translated and synthetic test distributions reflect real clinical text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the non-English test sets come from the same translated and synthetic distribution as the training data, so the 0.92–0.98 macro F1 scores are best read as upper bounds until the models are evaluated on authentic clinical notes written in those languages.
  • Editorial: because structured labels such as ZIP, PHONE, and SSN are handled by a rule-based layer, part of the reported F1 may reflect pattern matching rather than learned NER; ablating that layer would quantify its contribution.
  • Editorial: the GPT-4o comparison is English-only, so the claim that small models surpass general-purpose LLMs remains unverified for the seven non-English languages.
  • Editorial: a direct extension would be to run the same pipeline on existing public multilingual clinical corpora, which would test whether the data-generation strategy transfers beyond the authors' own datasets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes expert small NER models for PHI de-identification in eight languages, built using an LLM-in-the-loop methodology in which LLMs generate or translate synthetic training data and provide labels at intermediate stages. For English, the models are trained on a mix of open-source, synthetic, and proprietary data and evaluated on the external i2b2-2014 test set. For the seven non-English languages, training and test data are both taken as an 80-20 split of the same translated and synthetic corpus. The authors report macro-F1 scores around 0.92-0.96 on the non-English test splits and claim their models surpass prior small models and GPT-4o, with the abstract stating f1-micro averages between 0.953 and 0.978.

Significance. If the multilingual results were independently validated, the approach would be practically valuable because small, on-premises de-identification models avoid the privacy and API-dependency concerns of large closed models. The English evaluation is a genuine strength: the i2b2-2014 test set is an external, expert-annotated benchmark, and the strict chunk-plus-label matching is an appropriate evaluation protocol. The paper is also transparent in admitting that no non-English benchmark exists. That transparency, however, exposes a load-bearing weakness: the non-English evaluation is effectively a self-evaluation, so the headline cross-lingual claims are not currently supported.

major comments (3)
  1. [Sections 3.1, 3.2.2, 4; Tables 5-6] The non-English test sets are not independent of the training data. Section 3.1 states that most non-English datasets were generated by translating the English corpus with the authors' own medical translation models [30] and by LLM-assisted synthetic data generation, and Section 3.2.2 states that a 0.20 split of the same dataset is used as the test set. Section 4 then acknowledges that no benchmark tests exist for these languages. Consequently, the macro-F1 scores in Tables 5-6 measure fit to the same generated distribution from which the training data were drawn, not performance on real clinical notes in German, Italian, French, Romanian, Turkish, Spanish, or Arabic. The abstract's claim that the models are 'the most accurate healthcare anonymization solutions' across eight languages is therefore unsupported as stated. This requires external expert-annotated non-English test data, or a clear reframing of the non-English results as an internal pilot rather than a validated cross-lingual result.
  2. [Abstract; Tables 5-6] The headline numbers are not auditable. The abstract reports 'f1-micro score averages' of 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 for the eight languages, but Tables 5 and 6 report only macro-avg values (German 0.960, Italian 0.955, French 0.937, Turkish 0.963, Spanish 0.957, Romanian 0.930, Arabic 0.922). No micro-averaged F1 values are reported anywhere in the paper, and the abstract numbers do not match the presented macro-avg values. The authors should report the actual micro-averaged F1 used to generate the abstract, or correct the abstract.
  3. [Table 4, Section 4, Table 7] The GPT-4o comparison is internally inconsistent. Table 4 lists GPT-4o's macro-average as 0.548, but the Section 4 text says 'the macro average (0.5757)' and 'the micro average (0.5907)', while Table 7 reports a macro avg of 0.5775 and a weighted avg of 0.968. These discrepancies make it difficult to verify the claim that the proposed small models surpass GPT-4o. The paper should present one consistent evaluation of GPT-4o and report the exact numbers that support the comparison.
minor comments (4)
  1. [Section 3.2.2; Table 6] The Arabic model is included in Table 6, but Section 3.2.2 does not specify which embedding was used for Arabic, unlike the other languages. Please provide the Arabic embedding and its training details.
  2. [Table 7] The text following Table 7 mentions B-EMAIL and B-FAX as classes with low precision and recall, but the table does not contain rows for these labels. The caption and the table should be consistent.
  3. [Section 3.1; Table 4] The comparison with prior work in Table 4 uses a different label set: the authors include DEVICE and LOCATION-OTHER, which are absent from the comparator columns. Because macro-averaging gives equal weight to every class, the comparison is not strictly apples-to-apples; a footnote should state which labels are excluded from each comparator.
  4. [Footnote 1] The paper says the codebase will be available soon, but no code, data-generation script, or annotation protocol is released. Given the central role of synthetic data and LLM-assisted labeling, a detailed data-generation protocol is needed for reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Non-English F1 claims are measured on a holdout of the same self-generated translated/synthetic corpus, so the cross-language 'most accurate' claim is in-distribution validation, not independent prediction.

  1. fitted input called prediction [Section 3.1 (Datasets), Section 3.2.2 (Non-English de-identification Models), Section 4 (Result)]
    "Most of the non-English datasets were generated through translation from the English dataset using our medical translation models [30], open-source and through synthetic data generation with LLM-assisted labelling, producing high-quality parallel datasets across multiple languages. ... In the study, the 0.20 parts of the dataset determined during the division process were used as the test dataset. ... However, since it was impossible to find any benchmark tests for these languages, comparing the scores obtained in this study was impossible."

    The 80-20 split takes the non-English test set from the same self-generated corpus described in Section 3.1: the English source, the authors' own translation model [30], and LLM-assisted labelling produced both the training and the test distributions. A random split of one generated corpus cannot provide external validation because translation artifacts, synthetic templates, and LLM-labelling noise are shared between train and test. The near-ceiling macro-F1 scores in Tables 5-6 therefore measure in-distribution fit, not performance on real German, Turkish, Arabic, or other clinical notes.

full rationale

The English de-identification results are externally grounded: the model is tested on the i2b2-2014 test set, so the comparison with GPT-4o and prior small models in Table 4 is meaningful and not circular. The problem is the paper's central multilingual claim. Section 3.1 says most non-English datasets were produced by the authors' own Part-1 translation model [30] and by LLM-assisted synthetic generation; Section 3.2.2 then takes the test set as a 0.20 split of that same corpus. Because the training and test splits are random subsets of one generated distribution, the near-ceiling macro-F1 scores in Tables 5-6 measure fit to translation artifacts, synthetic templates, and LLM-labelling conventions shared by train and test. The paper explicitly states that no benchmark tests exist for these languages, so there is no external anchor for the 'across eight languages' result. This is a partial circularity: the source of the problem is the self-generated validation for the non-English claim, not the externally benchmarked English claim. Separately, the abstract reports 'f1-micro score averages' while Tables 5-6 show macro-avg values, which makes the headline numbers not directly auditable, though that is a reporting inconsistency rather than a circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on domain assumptions about label transfer, translation fidelity, and labelling quality. No new theoretical entities are introduced. The free parameters are standard training hyperparameters plus the split ratio, none of which are derived from first principles.

free parameters (5)
  • learning_rate = 2e-5
    Chosen by hand for fine-tuning; standard value, not central to the scientific claim.
  • batch_size = 2 for English and Romanian, 16 for others
    Chosen by hand; depends on GPU memory.
  • max_sentence_length = 512
    Chosen by hand to match transformer limits.
  • epochs = 10
    Chosen by hand; no early stopping or selection procedure reported.
  • non_english_train_test_split = 80/20
    Chosen by hand; test set is a random split of the self-generated corpus, which drives the evaluation.
assumptions (3)
  • domain assumption The i2b2 labeling logic applies unchanged to non-English clinical text.
    Section 3.1 states labels follow i2b2 2014 principles for all languages, assuming the same PHI categories and definitions transfer across languages.
  • domain assumption Machine-translated clinical text preserves PHI entity boundaries and label distributions.
    Section 3.1: non-English datasets were generated by translating English data with the authors' medical translation models [30]; the model is trained and evaluated on this translated text.
  • domain assumption LLM-assisted labelling produces ground-truth-quality labels for synthetic data.
    Section 3 says synthetic data was annotated by the labelling team with LLM assistance; no inter-annotator agreement or validation against real non-English notes is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages." pith.science (2026). https://pith.science/paper/NWBLL35Q

@misc{pith2026241210918,
  author       = {Pith},
  title        = {Pith review of: LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWBLL35Q}},
  note         = {Machine review of arXiv:2412.10918}
}
read the original abstract

The rise of chronic diseases and pandemics like COVID-19 has emphasized the need for effective patient data processing while ensuring privacy through anonymization and de-identification of protected health information (PHI). Anonymized data facilitates research without compromising patient confidentiality. This paper introduces expert small AI models developed using the LLM-in-the-loop methodology to meet the demand for domain-specific de-identification NER models. These models overcome the privacy risks associated with large language models (LLMs) used via APIs by eliminating the need to transmit or store sensitive data. More importantly, they consistently outperform LLMs in de-identification tasks, offering superior performance and reliability. Our de-identification NER models, developed in eight languages (English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic) achieved f1-micro score averages of 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 respectively. These results establish them as the most accurate healthcare anonymization solutions, surpassing existing small models and even general-purpose LLMs such as GPT-4o. While Part-1 of this series introduced the LLM-in-the-loop methodology for bio-medical document translation, this second paper showcases its success in developing cost-effective expert small NER models in de-identification tasks. Our findings lay the groundwork for future healthcare AI innovations, including biomedical entity and relation extraction, demonstrating the value of specialized models for domain-specific challenges.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 42 canonical work pages

  1. [30]

    LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation

    Keles, B., M. Gunay, and S.I. Caglar, LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation, arXiv preprint arXiv:2407.12126, 2024

    work page Pith review arXiv 2024

  2. [1]

    Aziz, and N

    Ahmed, T., M.M.A. Aziz, and N. Mohammed, De-identification of electronic health record using neural network, Sci Rep, 2020, 10(1): p. 18600

    work page 2020

  3. [2]

    Wood, A., et al., Linked electronic health records for research on a nationwi de cohort of more than 54 million people in England: data resource , BMJ, 2021, 373: p. n826

    work page 2021

  4. [3]

    Orhan, and N

    Gungoren, M., F. Orhan, and N. Kurutkan, Mikro Rekabetc ¸ilikte Yeni Yaklas ¸ımlar: Hastanelerde Olus ¸an Etik ˙Iklimin Kalite ve Akreditasyon Ac ¸ısından De ˘gerlendirilmesi, S¨ uleyman Demirel¨Universitesi ˙ Iktisadi ve ˙ Idari Bilimler Fak¨ ultesi Dergisi, 2013, 18(1): p. 221-241

    work page 2013

  5. [4]

    V arol, S ¸ ., et al., Sa˘glık kurumlarında bilgi g ¨uvenli˘gi ba ˘glamında biyometrik sistemler , Sa˘ glık Akademisyenleri Dergisi, 2016, 3(4): p. 155-162

    work page 2016

  6. [5]

    Erguner Ozkoc, and G

    Yilmaz, D., E. Erguner Ozkoc, and G. Ogutcu Ulas, Elektronik Sa ˘glık Kayıtlarında Farkındalık, 24, 2023

    work page 2023

  7. [6]

    healthITSecurity, De-Identification of PHI According to the HIPAA Privacy Rule , 2023, April 13, 2023; Available from: https://healthitsecurity .com/features/de-identification-of- phi-according-to-the-hipaa-privacy-rule

    work page 2023

  8. [7]

    Act, A., Health insurance portability and accountability act of 199 6, Public law, 1996, 104: p. 191

    work page 1996

Show all 47 references
  1. [8]

    Fernandez-Aleman, J.L., et al., Security and privacy in electronic health records: a sys- tematic literature review, J Biomed Inform, 2013, 46(3): p. 541-62

    2013

  2. [9]

    Final rule, Federal register, 2002, 67(157): p

    Office for Civil Rights, H., Standards for privacy of individually identifiable health i nfor- mation. Final rule, Federal register, 2002, 67(157): p. 53181-53273

    2002

  3. [10]

    Toscano, F., et al., Electronic health records implementation: can the Europea n Union learn from the United States? , European Journal of Public Health, 2018, 28(suppl 4): p. cky213. 401

    2018

  4. [11]

    hhs.gov, Guidance on De-identification of Protected Health Informat ion - hhs deid guidance.pdf, 2012; [cited 2023 July 17]; Available from: https://www.hhs.gov/sites/default/files/ocr/privacy/ hipaa/understanding/coveredentities/De- identification/hhs deid guidance.pdf

    2012

  5. [12]

    hhs.gov, Standards for Privacy of Individually Identifiable Health I nfo — HHS.gov , 2013; [cited 2023 July 17]; Available from: https://www.hh s.gov/hipaa/for- professionals/privacy/guidance/standards-privacy-individually-identifiable-health- information/index.html. 15

    2013

  6. [13]

    Neamatullah, I., et al., Automated de-identification of free-text medical records , BMC Med Inform Decis Mak, 2008, 8: p. 32

    2008

  7. [14]

    Paul, T., et al., Investigation of the Utility of Features in a Clinical De-id entification Model: A Demonstration Using EHR Pathology Reports for Adva nced NSCLC Patients , Front Digit Health, 2022, 4: p. 728922

    2022

  8. [15]

    Garfinkel, S., De-identification of Personal Information , 2015: US Department of Com- merce, National Institute of Standards and Technology

    2015

  9. [16]

    Wu, H., et al., SemEHR: A general-purpose semantic search system to surfac e semantic data from clinical notes for tailored care, trial recruitme nt, and clinical research , J Am Med Inform Assoc, 2018, 25(5): p. 530-537

    2018

  10. [17]

    Stubbs, A. and O. Uzuner, Annotating risk factors for heart disease in clinical narra tives for diabetic patients , J Biomed Inform, 2015, 58 Suppl(Suppl): p. S78-S91

    2015

  11. [18]

    19097- 19110

    Catelli, R., et al., A Novel COVID-19 Data Set and an Effective Deep Learning Appr oach for the De-Identification of Italian Medical Records , IEEE Access, 2021, 9: p. 19097- 19110

    2021

  12. [19]

    Reddy, S., et al., A governance model for the application of AI in health care , J Am Med Inform Assoc, 2020, 27(3): p. 491-497

    2020

  13. [20]

    Ong, J.C.L., et al., Artificial intelligence, ChatGPT, and other large language models for social determinants of health: Current state and future dir ections, Cell Rep Med, 2024, 5(1): p. 101356

    2024

  14. [21]

    e124-e134

    Gunasekeran, D.V ., et al., Digital health during COVID-19: lessons from operationali sing new models of care in ophthalmology , Lancet Digit Health, 2021, 3(2): p. e124-e134

    2021

  15. [22]

    Ting, D.S.W., et al., Digital technology and COVID-19, Nat Med, 2020, 26(4): p. 459-461

    2020

  16. [23]

    V erdicchio, M. and A. Perin, When Doctors and AI Interact: on Human Responsibility for Artificial Risks , Philos Technol, 2022, 35(1): p. 11

    2022

  17. [24]

    Xiong, and L.-W

    Dai, S.-C., A. Xiong, and L.-W. Ku, LLM-in-the-loop: Leveraging large language model for thematic analysis , arXiv preprint arXiv:2310.15100, 2023

    2023 arXiv

  18. [25]

    De Paoli, S., Can Large Language Models emulate an inductive Thematic Ana lysis of semi-structured interviews? An exploration and provocati on on the limits of the approach and the model , arXiv preprint arXiv:2305.13014, 2023

    2023 arXiv

  19. [26]

    Alizadeh, and M

    Gilardi, F., M. Alizadeh, and M. Kubli, ChatGPT outperforms crowd workers for text- annotation tasks, Proc Natl Acad Sci U S A, 2023, 120(30): p. e2305016120

    2023

  20. [27]

    Islam, T. and D. Goldwasser, Discovering latent themes in social media messaging: A machine-in-the-loop approach integrating llms, arXiv preprint arXiv:2403.10707, 2024

    2024 arXiv

  21. [28]

    Pham, D.K. and B.Q. V o, Towards Reliable Medical Question Answering: Tech- niques and Challenges in Mitigating Hallucinations in Lang uage Models, arXiv preprint arXiv:2408.13808, 2024. 16

    2024 arXiv

  22. [29]

    Roberts, and L

    Umphrey, R., J. Roberts, and L. Roberts, Investigating Expert-in-the-Loop LLM Discourse Patterns for Ancient Intertextual Analysis , arXiv preprint arXiv:2409.01882, 2024

    2024 arXiv

  23. [31]

    Burckhardt, and R

    Khin, K., P . Burckhardt, and R. Padman, A Deep Learning Architecture for De- identification of Patient Notes: Implementation and Evalua tion, arXiv pre-print server, 2018

    2018

  24. [32]

    Sengupta, and G

    Morrison, F.P ., S. Sengupta, and G. Hripcsak, Using a pipeline to improve de-identification performance, AMIA Annu Symp Proc, 2009. 2009: p. 447–51

    2009

  25. [33]

    Kotfila, and O

    Stubbs, A., C. Kotfila, and O. Uzuner, Automated systems for the de-identification of lon- gitudinal clinical narratives: Overview of 2014 i2b2/UTHe alth shared task Track 1 , J Biomed Inform, 2015, 58 Suppl(Suppl): p. S11-S19

    2014

  26. [34]

    Luo, and P

    Uzuner, O., Y . Luo, and P . Szolovits, Evaluating the state-of-the-art in automatic de- identification , J Am Med Inform Assoc, 2007, 14(5): p. 550–63

    2007

  27. [35]

    Dernoncourt, F., et al., De-identification of patient notes with recurrent neural ne tworks, J Am Med Inform Assoc, 2017, 24(3): p. 596–606

    2017

  28. [36]

    Ferrandez, O., et al., Evaluating current automatic de-identification methods wi th V et- eran’s health administration clinical documents , BMC Med Res Methodol, 2012, 12: p. 109

    2012

  29. [37]

    Meystre, S.M., et al., Automatic de-identification of textual documents in the ele ctronic health record: a review of recent research , BMC Med Res Methodol, 2010, 10: p. 70

    2010

  30. [38]

    Liu, Z., et al., Automatic de-identification of electronic medical records using token-level and character-level conditional random fields , J Biomed Inform, 2015, 58 Suppl(Suppl): p. S47-S52

    2015

  31. [39]

    Y ang, H. and J.M. Garibaldi, Automatic detection of protected health information from clinic narratives, J Biomed Inform, 2015, 58 Suppl(Suppl): p. S30-S38

    2015

  32. [40]

    Ohno-Machado, and W.W

    Nadkarni, P .M., L. Ohno-Machado, and W.W. Chapman, Natural language processing: an introduction, J Am Med Inform Assoc, 2011, 18(5): p. 544-51

    2011

  33. [41]

    Sweeney, L., Replacing personally-identifying information in medical records, the Scrub system, Proc AMIA Annu Fall Symp, 1996: p. 333-7

    1996

  34. [42]

    Saul, and J

    Gupta, D., M. Saul, and J. Gilbertson, Evaluation of a deidentification (De-Id) software engine to share pathology reports and clinical documents fo r research, Am J Clin Pathol, 2004, 121(2): p. 176-86

    2004

  35. [43]

    He, B., et al., CRFs based de-identification of medical records , J Biomed Inform, 2015, 58 Suppl(Suppl): p. S39-S46

    2015

  36. [44]

    McCallum, and F

    Lafferty, J., A. McCallum, and F. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data , in Icml. 2001. Williamstown, MA. 17

    2001

  37. [45]

    Talby, and H.U

    Kocaman, V ., D. Talby, and H.U. Hak, Beyond Accuracy: Automated De-Identification of Large Real-World Clinical Text Datasets, V alue in Health, 2023, 26(12): p. S532

    2023

  38. [46]

    Liu, Z., et al., Deid-gpt: Zero-shot medical text de-identification by gpt- 4, arXiv preprint arXiv:2303.11032, 2023

    2023

  39. [47]

    88 years old ,

    Stubbs, A., et al., Identifying risk factors for heart disease over time: Overv iew of 2014 i2b2/UTHealth shared task Track 2, J Biomed Inform, 2015, 58 Suppl(Suppl): p. S67-S77. 18 Appendix A- The Prompt Used to Obtain Benchmarks with GPT-4o 1 prompt = f """ You are tasked wi...

    2014

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.