Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

GENIE: Generative Note Information Extraction model for structuring EHR data

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GENIE, a fine-tuned 8-billion-parameter language model, structures raw clinical paragraphs into JSON records of medical terms and six attributes in a single pass, with reported accuracies that beat established tools on a MIMIC-III test set.

desk verdict A practical single-model clinical IE system with a genuinely useful integration, but the evaluation is too thin to support the headline accuracy claims. read the letter →

arxiv 2501.18435 v1 pith:FQAYJUK2 submitted 2025-01-30 cs.CL

Huaiyuan Ying , Hongyi Yuan , Jinsen Lu , Zitian Qu , Yang Zhao , Zhengyun Zhao , Isaac Kohane , Tianxi Cai
show 1 more author
Sheng Yu
This is my paper · ORCID
classification cs.CL
keywords electronichealthrecordsinformationextractiongenerativelanguagemodelclinicalNLPassertionstatusterminologynormalizationdatastructuringMIMIC-III
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single small open-source language model can replace the usual multi-module pipeline for structuring electronic health records. The model, GENIE, is a fine-tuned Llama-3.1-8B that reads a raw clinical paragraph and outputs a JSON list of medical terms, each with its semantic type, assertion status, locations, modifiers, values, units, and purposes in one pass. On a small human-labelled test set from MIMIC-III, it reports a phrase F1 of 0.837 and attribute accuracies from 0.812 to 0.934, outperforming traditional cTAKES and MetaMap on phrase extraction. The practical advantage is that hospitals could run this locally on consumer GPUs, avoiding the cost and privacy concerns of calling large proprietary models.

What carries the argument

The central object is the single-pass text-to-JSON formulation: all six attributes plus the entity mention are cast as one conditional generation problem, so a single decoder produces the whole structured output from raw text. This works because the training pipeline creates entity-wise records by matching GPT-annotated attributes to terms recognized by forward maximum matching against the BIOS ontology, and the model learns to reproduce that JSON structure, including automatic restoration of abbreviations and disambiguation by context.

What would settle it

Take a larger random sample of MIMIC-III notes (for example 200 paragraphs), have at least two independent clinical annotators label all phrases and attributes, and recompute GENIE's phrase F1 and attribute accuracies using exact string matching plus a pre-specified normalization table. If the numbers fall well below 0.837, 0.867, 0.934, 0.820, 0.812, and 0.912, the central claim would be contradicted.

Watch

Extended reading notes

Core claim

GENIE shows that one generative model can handle end-to-end EHR structurization: the same model performs named entity recognition, abbreviation restoration, assertion detection, and attribute extraction simultaneously. It is trained on 180,000 paragraphs produced by a pipeline that combines trie search against the BIOS ontology, a Llama-2-7b assertion classifier trained on the i2b2/VA challenge data, and GPT-generated location, modifier, value, unit, and purpose annotations. At inference, the raw note is fed directly to the model without task-specific instructions, and the model outputs the full structured record as JSON. The authors report phrase-level F1 of 0.837, location accuracy 0.867, modifier accuracy 0.934, value accuracy 0.820, unit accuracy 0.812, and assertion status accuracy 0.912 on 24 manually annotated paragraphs containing 448 phrases, against cTAKES at 0.182 and MetaMap at 0.172 phrase F1.

Load-bearing premise

The reported accuracies rest on just 24 hand-annotated paragraphs (448 phrases), with one annotator per paragraph and no cross-check, and on GPT-4o's word-equivalence judgments that could favour GENIE's GPT-trained output style.

Editorial extensions

If this is right

  • Replaces multiple modules (NER, assertion, location, and others) with one model, simplifying system maintenance and upgrades.
  • Enables local, low-cost processing on a 24GB GPU, making it feasible for institutions that cannot use proprietary cloud LLMs.
  • Reduces processing time by eliminating long instructional prompts and multi-stage calls.
  • Could extend to other attributes, note types, and low-resource languages; a Chinese version is noted as already developed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported numbers hold under independent adjudication, a generative model may capture clinical context (distant locations, acronym senses) that rule-based pipelines miss, which would argue for generative approaches to clinical information extraction generally.
  • The evaluation's dependence on GPT-4o equivalence judgments deserves a direct test: rerun scoring with exact-match normalization or human adjudication to see whether attribute accuracies change substantively.
  • The unmeasured 'purpose' attribute could be made measurable with a rubric-based evaluation, strengthening future claims about full attribute coverage.
  • The training-data reliance on GPT-generated attributes creates a potential feedback loop; monitoring for systematic attribute biases, such as location dislocations, would be a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces GENIE, an end-to-end system based on a fine-tuned Llama-3.1-8B-Instruct model that takes raw EHR paragraphs as input and outputs JSON-format extractions of medical phrases, assertion statuses, and attributes (locations, modifiers, values, units, and purposes). The training pipeline uses GPT models to annotate abbreviations, locations, modifiers, values, units, and purposes from MIMIC-III discharge notes, along with a separately trained Llama-2-7b model for assertion status. The model processes whole paragraphs in a single pass without task-specific prompts. The authors evaluate GENIE on a human-annotated test set of 24 paragraphs containing 448 phrases, reporting phrase F1 of 0.837, location accuracy 0.867, modifier accuracy 0.934, value accuracy 0.820, unit accuracy 0.812, and assertion status accuracy 0.912, and compare against cTAKES and MetaMap. They explicitly state that the purpose attribute is not numerically evaluated.

Significance. If the results hold up, GENIE would be a practical contribution: an open-source, locally deployable model that performs multi-attribute clinical information extraction in one pass, addressing a real gap in EHR structuring tools. The authors open-source the model and test data, which is valuable for reproducibility and follow-up work. The use of GPT-generated training labels and a fine-tuned small LLM is a pragmatic approach with potential for scalability. The comparison to cTAKES and MetaMap, while limited, indicates a large improvement on phrase extraction and assertion status on this test set. However, the significance is tempered by the evaluation weaknesses described below, which currently prevent the central accuracy claims from being fully established.

major comments (4)
  1. [Results, Test Datasets and attribute evaluation] The attribute accuracies for locations (0.867), modifiers (0.934), values (0.820), and units (0.812) are computed by having GPT-4o judge semantic equivalence between the model's predicted attribute expression and the gold-standard expression. This GPT-based equivalence judgment is never calibrated against human raters, and its potential bias is not analyzed. Because GENIE's training labels for these attributes were themselves produced by GPT models, GENIE's output phrasing is likely GPT-like; a GPT-based judge may systematically prefer such phrasing over the human gold expression. The paper's anecdotal claim that 'the reported metrics may be slightly underestimated' (Results, second discussion point) does not resolve this concern; without a human-equivalence calibration study, the direction and magnitude of any bias are unknown. I ask the authors to either (a) report a human-evaluation study on a sample of attribute predictions comparing GPT-4o equivalence judgments to human equivalence judgments, or (b) use exact-match or normalized-string matching (e.g., after unit normalization) as the primary metric, with GPT-4o equivalence as a secondary, clearly labeled sensitivity analysis.
  2. [Results, Test Datasets; Abstract] The abstract claims GENIE extracts 'purposes with high accuracy,' but the Results section explicitly states: 'The correctness of the purpose attribute was not numerically evaluated, as its narrative nature makes it unsuitable for the verification process.' The Discussion also asserts, without quantitative support, that 'the ability of this attribute is good.' This is a direct mismatch between a headline claim and the evidence presented. The authors should either provide a quantitative evaluation of the purpose attribute (e.g., human-annotated purposes on the test set, or a clearly defined proxy evaluation) or revise the abstract and conclusions to reflect that purpose extraction is demonstrated qualitatively only.
  3. [Results, Test Datasets] The test set is very small and lacks inter-annotator agreement: 24 paragraphs, 448 phrases, with each paragraph assigned to one of two annotators and no overlap to compute IAA. This raises concerns about the stability of the gold standard and the precision of all reported numbers. No confidence intervals or error bars are given. For example, the location accuracy of 0.867 on 235 location-bearing phrases has a 95% confidence interval of roughly ±0.04, and the modifier accuracy of 0.934 on 128 phrases has a CI of roughly ±0.06; the value/unit accuracies are based on 144 phrases. The authors should either enlarge the test set, or report confidence intervals and discuss the implications for the comparison against cTAKES and MetaMap. At a minimum, the small sample size and the single-annotation design must be acknowledged in the limitations section, which currently only lists model-error patterns and does not mention the evaluation's statistical fragility.
  4. [Results, Table 1 and Abstract] The claim that GENIE 'outperforming traditional tools like cTAKES and MetaMap' is only supported for the tasks those tools can actually perform: phrase extraction (F1: 0.837 vs 0.182 and 0.172) and assertion status (0.912 vs 0.748 for cTAKES). For locations, modifiers, values, and units, the table shows '-' because cTAKES and MetaMap do not extract these attributes, so the comparison is incomplete. The abstract's phrasing 'competitive performance across multiple information extraction tasks, outperforming traditional tools' should be narrowed to specify that the outperformance is on the shared tasks. This is not a fatal issue, but it is an overstatement that should be corrected.
minor comments (5)
  1. [Results, Test Datasets] The sentence 'manually annotated by two expert annotators, with each paragraph assigned to one annotator' is ambiguous; it should explicitly state that there was no overlap between annotators, which is why no inter-annotator agreement is reported. If any paragraphs were double-annotated for quality control, that should be described.
  2. [Methods, Assertion status] The rule-based additions ('allergy to conditional' and 'Title' detection) are ad hoc and not evaluated separately. Since these rules are applied to the training data, their effect on the final model's behavior should be discussed, especially the known bias toward 'present' mentioned in the Results.
  3. [References] Reference 10 and Reference 18 are the same work (Chapman et al., 2001) and are cited with identical titles; please merge them or cross-reference.
  4. [Results, second discussion point] The statement 'the reported metrics may be slightly underestimated' is presented without supporting evidence or analysis. If this is an observation from the authors, please provide a specific example or quantify how often equivalence judgments were false negatives in a hand-checked sample.
  5. [Abstract] The phrase 'with high accuracy' is used for the attributes, but the measured phrase F1 of 0.837 and value/unit accuracies around 0.81 are moderate. Consider using more precise language such as 'competitive accuracy' and report the quantitative ranges in the abstract or conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline comparison to cTAKES/MetaMap rests on a human-labelled exact-match phrase F1, and attribute accuracies use a human gold standard with GPT-4o only as a soft-match judge.

full rationale

GENIE's central claim of outperforming cTAKES and MetaMap is supported by phrase-level F1 on a human-annotated test set (448 phrases from 24 paragraphs) using exact-match scoring; this metric does not depend on the LLM family used to generate training labels. Attribute accuracies for location, modifier, value, and unit are measured against the same human gold standard, with GPT-4o used only to judge semantic equivalence of surface forms; while this judge is uncalibrated and could in principle favor GPT-style outputs, that is a measurement-validity limitation, not a reduction of the prediction to its training input. The paper explicitly omits numerical evaluation of 'purpose' (Results: 'The correctness of the purpose attribute was not numerically evaluated'), which weakens the abstract's blanket 'high accuracy' claim but is a missing-evidence issue, not circularity. Self-citations (BIOS, NILE) are tool/background references and are not used to justify performance claims. No fitted parameter is renamed as a prediction, and no result is forced by a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The model depends on several hand-chosen rules and unverified LLM annotations. The most load-bearing is the assumption that GPT-generated training labels are reliable, since the model effectively imitates the teacher's style, and that the small human test set plus GPT-based equivalence scoring provide an unbiased measure of performance.

free parameters (4)
  • allergy_to_conditional_rule = N/A
    In 'Assertion status', the authors convert the assertion status of all terms following the word 'allergy' to Conditional by rule, a hand-chosen post-processing step that adjusts for observed errors in the i2b2-derived model.
  • title_detection_rule = N/A
    A new 'Title' category is introduced and assigned to terms containing capital letters and immediately followed by ':', another ad hoc rule to handle formatting artifacts.
  • semantic_type_whitelist = 17 semantic types
    The output terms are filtered to a hand-selected list of 17 semantic types, which determines which entities are included in the training data and thus in the model's output.
  • section_length_recommendation = 800 tokens
    The authors recommend keeping sections shorter than 800 tokens to balance input length and output completeness, an engineering choice that affects the model's practical usage but not the central claim directly.
assumptions (4)
  • domain assumption MIMIC-III discharge notes are representative of general EHR narrative text
    The model is trained and tested only on MIMIC-III; no external validation on other datasets is performed, so the generalization to other EHR settings is assumed.
  • domain assumption GPT-generated annotations for locations, modifiers, values, units, and purposes are sufficiently accurate to serve as training targets
    The authors state in 'Limitations and future work' that 'attributes like locations and purposes are primarily generated by GPT, making it impractical to verify the correctness of each training data entry.' The model's quality is bounded by this unverified annotation quality.
  • domain assumption The i2b2 assertion model generalizes to MIMIC notes
    The assertion status model was fine-tuned on 2010 i2b2/VA challenge data and applied to MIMIC-III terms without additional adaptation, assuming the assertion categories match across corpora.
  • domain assumption Human annotations in the test set are correct
    The evaluation assumes the 448 manually labeled phrases are accurate, even though each paragraph was labeled by a single annotator and no inter-annotator agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GENIE: Generative Note Information Extraction model for structuring EHR data." pith.science (2026). https://pith.science/paper/FQAYJUK2

@misc{pith2026250118435,
  author       = {Pith},
  title        = {Pith review of: GENIE: Generative Note Information Extraction model for structuring EHR data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQAYJUK2}},
  note         = {Machine review of arXiv:2501.18435}
}
read the original abstract

Electronic Health Records (EHRs) hold immense potential for advancing healthcare, offering rich, longitudinal data that combines structured information with valuable insights from unstructured clinical notes. However, the unstructured nature of clinical text poses significant challenges for secondary applications. Traditional methods for structuring EHR free-text data, such as rule-based systems and multi-stage pipelines, are often limited by their time-consuming configurations and inability to adapt across clinical notes from diverse healthcare settings. Few systems provide a comprehensive attribute extraction for terminologies. While giant large language models (LLMs) like GPT-4 and LLaMA 405B excel at structuring tasks, they are slow, costly, and impractical for large-scale use. To overcome these limitations, we introduce GENIE, a Generative Note Information Extraction system that leverages LLMs to streamline the structuring of unstructured clinical text into usable data with standardized format. GENIE processes entire paragraphs in a single pass, extracting entities, assertion statuses, locations, modifiers, values, and purposes with high accuracy. Its unified, end-to-end approach simplifies workflows, reduces errors, and eliminates the need for extensive manual intervention. Using a robust data preparation pipeline and fine-tuned small scale LLMs, GENIE achieves competitive performance across multiple information extraction tasks, outperforming traditional tools like cTAKES and MetaMap and can handle extra attributes to be extracted. GENIE strongly enhances real-world applicability and scalability in healthcare systems. By open-sourcing the model and test data, we aim to encourage collaboration and drive further advancements in EHR structurization.

Figures

Figures reproduced from arXiv: 2501.18435 by the authors.

Figure 1
Figure 1. An example of our model rationale. Typical raw EHR notes contains plenty of jargons and acronyms, which can be restored for annotation process. When processing training data, we extract attributes and full-name phrases from the restored version, but the input will keep the raw version. The training process will lead our model to learn extract the understandable phrase, semantic types, assertion status, locations, mo… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment

    cs.IR 2025-02 conditional novelty 6.0 of 10

    CliniQ is a public EHR retrieval benchmark with 77,206 LLM-annotated relevance judgments, showing that BM25 is a strong baseline and that semantic matches drive dense-retriever gains.

Reference graph

Works this paper leans on

30 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    A data quality ontology for the secondary use of EHR data

    Johnson, Steven G., et al. "A data quality ontology for the secondary use of EHR data." AMIA Annual Symposium Proceedings. V ol. 2015. American Medical Informatics Association, 2015

    work page 2015

  2. [2]

    Recommendations for the use of operational electronic health record data in comparative effectiveness research

    Hersh, William R., et al. "Recommendations for the use of operational electronic health record data in comparative effectiveness research." EGEMS 1.1 (2013)

    work page 2013

  3. [3]

    Clinical information extraction applications: a literature review

    Wang, Yanshan, et al. "Clinical information extraction applications: a literature review." Journal of biomedical informatics 77 (2018): 34-49

    work page 2018

  4. [4]

    Impacts of structuring the electronic health record: Results of a systematic literature review from the perspective of secondary use of patient data

    Vuokko, Riikka, et al. "Impacts of structuring the electronic health record: Results of a systematic literature review from the perspective of secondary use of patient data." International journal of medical informatics 97 (2017): 293-303

    work page 2017

  5. [5]

    The evolving use of electronic health records (EHR) for research

    Kim, Ellen, et al. "The evolving use of electronic health records (EHR) for research." Seminars in radiation oncology. V ol. 29. No. 4. WB Saunders, 2019

    work page 2019

  6. [6]

    Efficient string matching: an aid to bibliographic search

    Aho, Alfred V ., and Margaret J. Corasick. "Efficient string matching: an aid to bibliographic search." Communications of the ACM 18.6 (1975): 333-340

    work page 1975

  7. [7]

    The unified medical language system (UMLS): integrating biomedical terminology

    Bodenreider, Olivier. "The unified medical language system (UMLS): integrating biomedical terminology." Nucleic acids research 32.suppl_1 (2004): D267-D270

    work page 2004

  8. [8]

    Bios: An algorithmically generated biomedical knowledge graph

    Yu, Sheng, et al. "Bios: An algorithmically generated biomedical knowledge graph." arXiv preprint arXiv:2203.09975 (2022)

    arXiv 2022

Show all 30 references
  1. [9]

    term: status\n

    supports only NER and entity linking, while cTAKES [16] and CLAMP [17] extend this to include assertion analysis. NILE [20] further integrates location and modification analyses. The interdependencies among these modules pose significant challenges, particularly for system mai...

    2010

  2. [11]

    Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program

    Aronson, Alan R. "Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program." Proceedings of the AMIA Symposium. American Medical Informatics Association, 2001

    2001

  3. [12]

    SNOMED -CT: The advanced terminology and coding system for eHealth

    Donnelly, Kevin. "SNOMED -CT: The advanced terminology and coding system for eHealth." Studies in health technology and informatics 121 (2006): 279

    2006

  4. [13]

    BERN2: an advanced neural biomedical named entity recognition and normalization tool

    Sung, Mujeen, et al. "BERN2: an advanced neural biomedical named entity recognition and normalization tool." Bioinformatics 38.20 (2022): 4837-4839

    2022

  5. [14]

    RxNorm: prescription for electronic drug information exchange

    Liu, Simon, et al. "RxNorm: prescription for electronic drug information exchange." IT professional 7.5 (2005): 17-23

    2005

  6. [15]

    ICD -11: an international classification of diseases for the twenty - first century

    Harrison, James E., et al. "ICD -11: an international classification of diseases for the twenty - first century." BMC medical informatics and decision making 21 (2021): 1-10

    2021

  7. [16]

    Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications

    Savova, Guergana K., et al. "Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications." Journal of the American Medical Informatics Association 17.5 (2010): 507-513

    2010

  8. [17]

    Current procedural terminology: CPT

    American Medical Association. "Current procedural terminology: CPT." (No Title) (1999).\

    1999

  9. [18]

    A simple algorithm for identifying negated findings and diseases in discharge summaries

    Chapman, Wendy W., et al. "A simple algorithm for identifying negated findings and diseases in discharge summaries." Journal of biomedical informatics 34.5 (2001): 301-310

    2001

  10. [19]

    CLAMP –a toolkit for efficiently building customized clinical natural language processing pipelines

    Soysal, Ergin, et al. "CLAMP –a toolkit for efficiently building customized clinical natural language processing pipelines." Journal of the American Medical Informatics Association 25.3 (2018): 331-336

    2018

  11. [20]

    NILE: fast natural language processing for electronic health records

    Yu, Sheng, Tianrun Cai, and Tianxi Cai. "NILE: fast natural language processing for electronic health records." arXiv preprint arXiv:1311.6063 (2013)

    2013 arXiv

  12. [21]

    ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports

    Harkema, Henk, et al. "ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports." Journal of biomedical informatics 42.5 (2009): 839-851

    2009

  13. [22]

    Extracting timing and status descriptors for colonoscopy testing from electronic medical records

    Denny, Joshua C., et al. "Extracting timing and status descriptors for colonoscopy testing from electronic medical records." Journal of the American Medical Informatics Association 17.4 (2010): 383-388

    2010

  14. [23]

    2010 i2b2/V A challenge on concepts, assertions, and relations in clinical text

    Uzuner, Özlem, et al. "2010 i2b2/V A challenge on concepts, assertions, and relations in clinical text." Journal of the American Medical Informatics Association 18.5 (2011): 552-556

    2011

  15. [24]

    DeepPhe: a natural language processing system for extracting cancer phenotypes from clinical records

    Savova, Guergana K., et al. "DeepPhe: a natural language processing system for extracting cancer phenotypes from clinical records." Cancer research 77.21 (2017): e115-e118

    2017

  16. [25]

    Chem 140, 3.6, 103, 26, 20, 1.1 and 107

    and NILE [20]. However, these tools require users to predefine the attributes to be extracted, rather than performing fully automated processing. For this reason, we exclude them from our baseline comparison, as our structuring process does not involve pre-specification of att...

  17. [26]

    Automated information extraction on treatment and prognosis for non – small cell lung Cancer radiotherapy patients: clinical study

    Zheng, Shuai, et al. "Automated information extraction on treatment and prognosis for non – small cell lung Cancer radiotherapy patients: clinical study." JMIR Medical Informatics 6.1 (2018): e8662

    2018

  18. [27]

    EXTraction of EMR numerical data: an efficient and generalizable tool to EXTEND clinical research

    Cai, Tianrun, et al. "EXTraction of EMR numerical data: an efficient and generalizable tool to EXTEND clinical research." BMC medical informatics and decision making 19 (2019): 1-7

    2019

  19. [28]

    MedEx: a medication information extraction system for clinical narratives

    Xu, Hua, et al. "MedEx: a medication information extraction system for clinical narratives." Journal of the American Medical Informatics Association 17.1 (2010): 19-24

    2010

  20. [29]

    Large language models encode clinical knowledge

    Singhal, Karan, et al. "Large language models encode clinical knowledge." Nature 620.7972 (2023): 172-180

    2023

  21. [30]

    PMC -LLaMA: toward building open -source language models for medicine

    Wu, Chaoyi, et al. "PMC -LLaMA: toward building open -source language models for medicine." Journal of the American Medical Informatics Association (2024): ocae045

    2024

  22. [31]

    MIMIC-III, a freely accessible critical care database

    Johnson, Alistair EW, et al. "MIMIC-III, a freely accessible critical care database." Scientific data 3.1 (2016): 1-9

    2016

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.