Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Asking an LLM to reason like a physician, then verify its own answer, extracts OPQRST symptoms from electronic health record notes at F1 up to 0.944, above five standard prompting baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Reasoning-style prompts improve few-shot LLM extraction of OPQRST items from EHR notes, but the result rests on an 85-note single-annotator evaluation with an LLM judge.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful but uncontrolled: the reported gains over baselines mix reasoning with few-shot demonstrations, so the headline F1 numbers should not be taken at face value. the 4 major comments →

arxiv 2509.01885 v1 pith:4WRANXZV submitted 2025-09-02 cs.CL cs.AI

Extracting OPQRST in Electronic Health Records using Large Language Models with Reasoning

Zhimeng Luo , Abhibha Gupta , Adam Frisch , Daqing He This is my paper
classification cs.CL cs.AI
keywords OPQRST extractionelectronic health recordslarge language modelsin-context learningnamed entity recognitionfew-shot promptingself-verificationclinical NLP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that OPQRST symptoms—Onset, Provocation/Palliation, Quality, Region/Radiation, Severity, Time—can be pulled out of electronic health record notes far more reliably by asking a large language model to generate the answer with reasoning than by tagging text spans as named entities. The authors build a six-part prompt for Llama-2-13B that makes the model first identify the chief complaint, then follow physician-style cues for each OPQRST entity, and finally re-check its own reasoning before answering; on their 85-note test set this reaches F1 of 0.944 for Onset against 0.618 for the best baseline style, and it leads on all seven entities. They also argue that exact-match evaluation is the wrong yardstick for generated clinical phrases, and that judging semantic equivalence—with BERTScore or with an LLM judge—better reflects clinical utility; their LLM judge agrees with human ratings at Cohen's kappa above 0.7 across entities. The practical stake is that structured, clinically meaningful symptom documentation could be recovered from unstructured notes with a handful of examples and no fine-tuning.

Core claim

The paper's central claim is that OPQRST extraction is best treated as structured text generation with explicit reasoning rather than as sequence labeling. Its six-part prompt defines the task and entities, supplies physician-style search cues (marker phrases such as 'presents with' for the chief complaint, temporal markers for Onset), gives a four-step reasoning template, adds three few-shot demonstrations covering present and absent cases, and instructs the model to re-read its reasoning before answering. The Onset ablation reports F1 dropping from 0.94 to 0.84 without the reasoning steps and to 0.77 without self-verification. On evaluation, the paper argues that an LLM judge deciding whet

What carries the argument

The load-bearing object is the six-part extraction prompt for Llama-2-13B-chat, and inside it the physician-mimicking reasoning template: identify the verb tense and marker phrase that introduce the chief complaint, name the complaint, locate the phrase that answers the OPQRST question, then emit the final answer between '@' delimiters so it can be parsed automatically. The self-verification step—asking the model to re-read its reasoning before committing—is the mechanism credited with suppressing hallucinations at the end of the chain. A second piece of machinery is the evaluation stack: exact-match F1 following the SemEval-2013 Task 9.1 protocol, extended with a semantic comparator (BERTSc

Load-bearing premise

The reported gains stand or fall on two conditions the paper leaves implicit: no few-shot demonstration note appears in the 85-note test set, and the LLM judge's match decisions agree with a clinician's—if either fails, the F1 numbers are optimistic.

What would settle it

Two concrete checks would settle it: scan the 85-note test set for verbatim strings from the demonstration HPI notes reprinted in the appendix (any overlap invalidates the baseline comparison), and have two independent physician annotators re-judge the same predictions the PromptLLM judge scored—if physicians do not agree with each other better than PromptLLM agrees with one physician, the kappa validation does not show the judge is faithful.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A few-shot prompt with no fine-tuning can recover all seven OPQRST entities from history-of-present-illness notes, which matters where labeled clinical data are scarce or private.
  • Explicit reasoning steps make extraction interpretable: a clinician can read why the model chose a phrase, and the self-verification step is a cheap guard against hallucinated answers.
  • Semantic evaluation changes what counts as a correct extraction, so paraphrases that preserve clinical meaning are no longer penalized as errors.
  • Because entity extraction is anchored on the chief complaint, the same template should generalize to other complaint-anchored clinical entities beyond OPQRST, as the paper's discussion states.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the chief-complaint anchor is what carries the gain, the template should transfer to other symptom mnemonics (PQRST for chest pain, OLDCARTS for pain history) by swapping entity definitions and marker phrases—an easy test needing no new annotation.
  • The evaluation protocol could be stress-tested by re-scoring predictions against paraphrased gold phrases; if the LLM judge is lenient, paraphrase-based golds would inflate agreement without any change in the extractor.
  • The 85-note test set comes from one annotation effort, so a multi-site replication would show whether the prompt's gains survive shifts in note style, dictation artifacts, and hospital-specific phrasing.
  • The self-verification step could be probed directly by injecting a fabricated reasoning link into the chain and checking whether the model catches it, separating genuine verification from re-affirming its own output.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper treats OPQRST extraction from EHR History of Present Illness notes as a text-generation task rather than sequence labeling. A Llama-2-13B-chat model is prompted with a task definition, entity definitions, physician-mimicking reasoning steps, a self-verification step, and three few-shot examples; outputs are bracketed with '@' tokens and evaluated by an LLM-based semantic judge (PromptLLM) or BERTScore. On an 85-sample test set of ED HPI notes, the proposed prompt achieves F1 0.944 for Onset versus 0.618 for the best baseline and improves all seven entities (Table 2). An ablation on Onset shows F1 drops from 0.94 to 0.84 without reasoning steps and to 0.769 without self-verification (Table 3). The authors also propose semantic-similarity extensions to exact-match NER metrics and discuss generalizable insights for low-resource clinical NLP.

Significance. If the reported gains are genuine, the paper offers a practical, few-shot alternative to sequence-labeling models in low-resource clinical settings and a reasonable way to soften exact-match evaluation for generative extractors. The ablation is a clear strength: it separately tests reasoning and self-verification, and the paper is honest about its limitations (Sec. 7). The main contribution, however, is an empirical claim, and its validity depends on controls that are currently missing: demonstration/test-set disjointness, matched in-context examples across baselines, and a judge that has been validated on all methods. With those controls, the result would be publishable; without them, the central comparison is not yet established.

major comments (4)
  1. [Table 2; Sec. 3.2; Appendix (Baseline Prompts)] The comparison is confounded by in-context demonstrations. The proposed prompt contains three few-shot examples (Sec. 3.2, part 5), while the Prefix, Cloze, Anticipatory, and Heuristic baseline prompts in the appendix contain no demonstrations; only COT contains one. Therefore the large gains (e.g., Onset 0.944 vs 0.618) may be due to the number of demonstrations rather than physician-style reasoning or self-verification. The ablation (Table 3) isolates reasoning/self-verification, but it does not give the same demonstrations to the Heuristic baseline. Add a matched control: the best zero-shot baseline plus the proposed three examples, and a no-example version of the proposed prompt. This is required to identify the claimed mechanism.
  2. [Appendix (Ablation Prompt 1); Sec. 4.1] The evaluation is valid only if the demonstration notes are disjoint from the 85 test samples. The appendix reuses a full HPI note ('73-year-old male ... chest pain') as Example1 in the ablation prompts, and the COT baseline prompt uses the same note. The paper never states whether this note or the other two examples are held out of the test set. Since the test set consists of HPI notes from the same ED, overlap is a real risk and would inflate all F1 numbers, especially Onset. Please report the intersection between demonstration notes and test samples, and rerun with disjoint demonstrations if any overlap exists.
  3. [Table 2; Sec. 4.2] Table 2 reports point estimates only, with no confidence intervals, standard errors, or significance tests. The model is sampled with temperature 1.0 and top-p 0.95 (Sec. 4.2), so output is stochastic. Some gaps are small (Quality 0.673 vs 0.603; Region 0.736 vs 0.667) and could be within sampling noise. Report bootstrap CIs over test samples or notes and paired significance tests across methods, or at least repeated decoding with different seeds. This is necessary to support the claim that the method outperforms all baselines on all entities.
  4. [Sec. 5, Table 1; Sec. 3.3] The primary metric is an LLM judge whose validity is not established for this purpose. Table 1 shows Cohen's kappa above 0.7 on 85 samples between PromptLLM and human evaluation, but the paper then abandons human evaluation for all remaining results. Moreover, the same class of LLM was used in Sec. 3.1 and Sec. 3.3 to refine both the extraction prompt and the evaluation prompt, so the judge may be biased toward the method's output phrasing and format. Report human F1 for all methods (or on a substantial random subset), provide the exact evaluation prompt and decision rule, and include exact-match F1 alongside semantic F1 in Table 2, so the judge's effect can be separated from true clinical-content gains.
minor comments (5)
  1. [Table 1] Typo: 'Kappa-BER TScore' should be 'Kappa-BERTScore'. Also clarify whether the kappa values are computed per entity on the 85 samples or on the per-entity subsets.
  2. [Sec. 3.3] The evaluation metric is described only at a high level. Since all downstream numbers use PromptLLM, provide the final evaluation prompt, the exact instruction for Yes/No, and the rule for mapping the model's output to phrase-level decisions. The current narrative in five steps is not reproducible.
  3. [Sec. 5.1, Figure 2] Figure 2 is referenced as showing prompt variants and prediction examples, but the appendix only gives 'Ablation prompt 1' and 'Ablation prompt 2'; the exact full proposed prompt and the self-verification step do not appear in the manuscript. Please add the full prompt text and clarify which parts of Figure 2 correspond to which ablation.
  4. [Sec. 3.1; Sec. 4.1] There is a dangling reference to a 'question decomposition paper' with no citation. Also, Sec. 4.1 says 85 samples, but because multi-mention annotations are replicated into new data points, the number of unique HPI notes underlying the 85 samples is not reported. This matters for the overlap check and for any bootstrap performed over notes.
  5. [General] Careful proofreading is needed: the author line has 'Adam F risch', Ablation prompt 1 has 'extract the the words', and several appendix examples contain grammatical errors. These are not substantive issues but should be fixed.

Circularity Check

0 steps flagged

No circular derivation: the paper's claims are empirical comparisons; the main threats are experimental-control and evaluator-validity concerns, not equation-level or self-citation circularity.

full rationale

The paper does not present a formal derivation in which an output is constructed from its own target. The central comparison (Section 5, Table 2) is an empirical evaluation on an 85-note physician-annotated test set, and the proposed method is not fitted to the test labels: its few-shot examples are fixed demonstrations, and the ablations (Table 3) vary reasoning and self-verification components rather than reusing the target metric as an input. The only self-citation, Luo et al. (2023), is used to motivate a change from sequence labeling to text generation and is not load-bearing. The use of PromptLLM as the primary evaluator after a kappa check against human scoring (Table 1) is an evaluation-validity concern, but it is not a reduction by construction or a fitted parameter renamed as a prediction. The uncontrolled comparison—three in-context demonstrations for the proposed method versus zero-shot baselines adapted from Sivarajkumar et al. (2024)—and the lack of an explicit holdout statement for the appendix's demonstration HPI note are experimental-control and missing-support concerns, not circular steps. No load-bearing self-citation chain, imported uniqueness theorem, or ansatz smuggled in via citation is present. Therefore no specific circular step can be exhibited under the required standard.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The quantitative comparison rests on four domain assumptions: chief-complaint dependency, single-physician ground truth, semantic similarity as a proxy for clinical equivalence, and disjointness of demonstrations from the test set. The last is unstated and critical. Sampling hyperparameters and the few-shot examples are hand-chosen and could fit the test set. No new physical entities are introduced.

free parameters (2)
  • Sampling hyperparameters = temperature=1.0, top_p=0.95, top_k=30
    Section 4.2 states these were selected based on experimentation and observing outputs; they are hand-chosen and could influence the reported F1 scores.
  • Few-shot demonstration examples = 3 examples for the main prompt, with one full HPI note reused in appendix prompts
    Section 3.2 says examples were carefully chosen by hand; the paper does not state whether they were excluded from the 85-note test set, so they may act as fitted examples.
axioms (4)
  • domain assumption OPQRS entities depend on the chief complaint, so extracting the chief complaint first improves entity extraction.
    Used in Section 3.1 and 3.2 to justify the two-stage prompt; not empirically validated in the paper.
  • domain assumption A physician's annotations of 85 HPI notes are valid ground truth, including empty strings for absent entities.
    Section 4.1 describes a single collaborating physician and annotation rules; no inter-annotator agreement is reported.
  • domain assumption BERTScore and PromptLLM semantic similarity capture clinical equivalence between generated and annotated phrases.
    Section 3.3 and Table 1 rely on this; only aggregate kappa against one human is reported, with no failure analysis.
  • ad hoc to paper The few-shot demonstration notes are disjoint from the 85 evaluation samples.
    Never stated in the paper, but required for Table 2 comparisons to be valid; the appendix reuses a full HPI note as a demonstration.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Extracting OPQRST in Electronic Health Records using Large Language Models with Reasoning." pith.science (2026). https://pith.science/paper/4WRANXZV

@misc{pith2026250901885,
  author       = {Pith},
  title        = {Pith review of: Extracting OPQRST in Electronic Health Records using Large Language Models with Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WRANXZV}},
  note         = {Machine review of arXiv:2509.01885}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The extraction of critical patient information from Electronic Health Records (EHRs) poses significant challenges due to the complexity and unstructured nature of the data. Traditional machine learning approaches often fail to capture pertinent details efficiently, making it difficult for clinicians to utilize these tools effectively in patient care. This paper introduces a novel approach to extracting the OPQRST assessment from EHRs by leveraging the capabilities of Large Language Models (LLMs). We propose to reframe the task from sequence labeling to text generation, enabling the models to provide reasoning steps that mimic a physician's cognitive processes. This approach enhances interpretability and adapts to the limited availability of labeled data in healthcare settings. Furthermore, we address the challenge of evaluating the accuracy of machine-generated text in clinical contexts by proposing a modification to traditional Named Entity Recognition (NER) metrics. This includes the integration of semantic similarity measures, such as the BERT Score, to assess the alignment between generated text and the clinical intent of the original records. Our contributions demonstrate a significant advancement in the use of AI in healthcare, offering a scalable solution that improves the accuracy and usability of information extraction from EHRs, thereby aiding clinicians in making more informed decisions and enhancing patient care outcomes.

Figures

Figures reproduced from arXiv: 2509.01885 by Abhibha Gupta, Adam Frisch, Daqing He, Zhimeng Luo.

Figure 1
Figure 1. Figure 1: Heatmap of OPQRS breakdown results [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Detailed prompts used in ablation study. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 10 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Large language models are few-shot clinical information extractors

    Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998--2022, 2022

    work page 2022

  3. [3]

    Comprehend medical: a named entity recognition and relationship extraction web service

    Parminder Bhatia, Busra Celikkaya, Mohammed Khalilia, and Selvan Senthivel. Comprehend medical: a named entity recognition and relationship extraction web service. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1844--1851. IEEE, 2019

    work page 2019

  4. [4]

    Generating contextual embeddings for emergency department chief complaints

    David Chang, Woo Suk Hong, and Richard Andrew Taylor. Generating contextual embeddings for emergency department chief complaints. JAMIA open, 3 0 (2): 0 160--166, 2020

    work page 2020

  5. [5]

    Medical information extraction in the age of deep learning

    Udo Hahn and Michel Oleynik. Medical information extraction in the age of deep learning. Yearbook of medical informatics, 29 0 (01): 0 208--220, 2020

    work page 2020

  6. [6]

    Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415, 2019

    Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415, 2019

    Pith/arXiv arXiv 1901

  7. [7]

    Few-shot learning for named entity recognition in medical text

    Maximilian Hofer, Andrey Kormilitzin, Paul Goldberg, and Alejo Nevado-Holgado. Few-shot learning for named entity recognition in medical text. arXiv preprint arXiv:1811.05468, 2018

    Pith/arXiv arXiv 2018

  8. [8]

    Information extraction from electronic medical documents: state of the art and future research directions

    Mohamed Yassine Landolsi, Lobna Hlaoua, and Lotfi Ben Romdhane. Information extraction from electronic medical documents: state of the art and future research directions. Knowledge and Information Systems, 65 0 (2): 0 463--516, 2023

    work page 2023

  9. [9]

    How far is language model from 100\ arXiv preprint arXiv:2307.00186, 2023

    Mingchen Li and Rui Zhang. How far is language model from 100\ arXiv preprint arXiv:2307.00186, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55 0 (9): 0 1--35, 2023

    2023

  12. [12]

    Towards accurate and clinically meaningful summarization of electronic health record notes: A guided approach

    Zhimeng Luo, Yuelyu Ji, Abhibha Gupta, Zhuochun Li, Adam Frisch, and Daqing He. Towards accurate and clinically meaningful summarization of electronic health record notes: A guided approach. In 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), pages 1--5, 2023. doi:10.1109/BHI58575.2023.10313411

    arXiv 2023

  13. [13]

    Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022

    2022

  14. [14]

    Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013)

    Isabel Segura Bedmar, Paloma Mart \' nez, and Mar \' a Herrero Zazo. Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). Association for Computational Linguistics, 2013

    work page 2013

  15. [15]

    An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: Algorithm development and validation study

    Sonish Sivarajkumar, Mark Kelley, Alyssa Samolyk-Mazzanti, Shyam Visweswaran, and Yanshan Wang. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: Algorithm development and validation study. JMIR Med Inform, 12: 0 e55318, Apr 2024. ISSN 2291-9694. doi:10.2196/55318. URL https://medi...

    doi:10.2196/55318 2024

  16. [16]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Gpt-ner: Named entity recognition via large language models

    Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428, 2023 a

    Pith/arXiv arXiv 2023

  18. [18]

    Clinical information extraction applications: A literature review

    Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, and Hongfang Liu. Clinical information extraction applications: A literature review. Journal of Biomedical Informatics, 77: 0 34--49, 2018. ISSN 1532-0464. doi:https://doi.org/10.1016/j.jbi.2017.11.011. URL https...

    doi:10.1016/j.jbi.2017.11.011 2018

  19. [19]

    Are large language models ready for healthcare? a comparative study on clinical language understanding

    Yuqing Wang, Yun Zhao, and Linda Petzold. Are large language models ready for healthcare? a comparative study on clinical language understanding. In Machine Learning for Healthcare Conference, pages 804--823. PMLR, 2023 b

    work page 2023

  20. [20]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824--24837. Curran Asso...

    work page 2022

  21. [21]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr

    work page 2020

  22. [22]

    A neural multi-task learning framework to jointly model medical named entity recognition and normalization

    Sendong Zhao, Ting Liu, Sicheng Zhao, and Fei Wang. A neural multi-task learning framework to jointly model medical named entity recognition and normalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 817--824, 2019

    work page 2019

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.