REVIEW 4 major objections 5 minor 1 cited by
Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Existing frontier models can already do universal medical abstraction, the paper argues, and a modular zero-shot prompt template is enough to expose it across clinical attributes.
desk verdict A solid, practically useful clinical NLP paper whose long-context wins are real but partly owe to GPT-4 summarizing the notes before the scoring model sees them—worth engaging, but cite the pipeline, not the 'single model' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the UMA prompt template: a reusable scaffold with a fixed AI-assistant role definition and general extraction instruction, a replaceable attribute-definition block that names the attribute, its descriptors, the expected output JSON format, and a per-patient input block. For long-context staging attributes, a one-time GPT-4 pass condenses hundreds of pages of clinical guidelines into attribute-specific instructions, and another pass condenses a patient's many notes into a chronological summary; the model is also asked to emit reasoning and evidence for each staging value, and previously extracted structured attributes such as tumor site and treatment date can be chained into the prompt. The template is what turns a general-purpose model into an attribute-specific extractor without training, and the evidence-and-reasoning descriptors are what make the outputs traceable and more accurate on complex staging tasks.
What would settle it
Re-run the long-context evaluation on the same test sets with the summarization step removed and with labels re-derived by independent human chart review alone; if UMA's average advantage over supervised baselines, about two points on the primary set and about seven points on the held-out set, does not survive, the universal-abstraction claim would be shown to depend on the pre-summarization or the label construction rather than on a property of frontier models.
Extended reading notes
Core claim
The paper's central claim is that "universal abstraction" is already present in frontier models: one model, prompted through a modular template, can structure a wide range of clinical attributes from raw patient notes without attribute-specific training. The evidence is a set of oncology attributes spanning simple single-note extractions, such as PD-L1, performance status, treatment, response, progression, and case finding, as well as long-context staging attributes that require reasoning over many notes and hundreds of pages of clinical guidelines. Against conventional baselines, UMA with GPT-4o matches or exceeds them overall by about two absolute points in F1 or accuracy on the primary dataset, including a gain of over twenty points on pathologic T staging; on a held-out second-institution dataset, UMA's average accuracy leads the supervised baseline by roughly seven points. The paper further claims that this is achieved at a fraction of the conventional development cost, with under an hour of prompt engineering replacing tens of thousands of hours of manual labeling.
Load-bearing premise
The comparison is only as clean as the labels and the inputs the model actually reads: the performance-status labels were produced by one clinician with model-assisted adjudication, and every long-context patient history was pre-summarized by GPT-4 before the evaluated model saw it.
Editorial extensions
If this is right
- Onboarding a new abstraction attribute reduces to writing an attribute definition block, so an attribute that once required months of labeling or rule crafting can be configured in under an hour.
- The same zero-shot prompt transfers across institutions: on the held-out second-institution dataset, UMA's average accuracy was about seven points above the supervised baseline that had been trained on the primary dataset.
- Guideline updates can be absorbed by rewriting prompt text rather than recollecting labels, which matters for staging systems that change over time.
- For long-context staging attributes, reasoning models such as O1 push accuracy further, so the framework's performance will track improvements in the underlying frontier model.
- Human oversight remains part of the workflow, but the built-in evidence and reasoning fields make model outputs checkable, which is what allows efficient clinical review.
Reading between the lines
- Beyond the paper's oncology testbed, the same template logic should extend to any attribute that can be specified in natural language, so a natural next test is whether the average gain persists for administrative, genomic, or non-oncology clinical attributes.
- Because long-context accuracy depends on GPT-4's pre-summarization of both guidelines and patient histories, the framework's ceiling is partly set by the summarizer; a testable corollary is that a more faithful summarizer should raise downstream staging accuracy even if the final reader model is unchanged.
- The performance-status ground truth involved model-assisted adjudication by a model from the same family as those being scored, so an independent human-only re-annotation would be the cleanest way to separate genuine universal abstraction from evaluation loop effects on that attribute.
- The framework is model-agnostic, so as open-weight models approach GPT-4o-level reasoning, UMA's cost advantage relative to supervised pipelines should grow; the paper leaves that comparison to future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents UniMedAbstractor (UMA), a zero-shot prompting framework for medical abstraction that extracts structured clinical attributes from unstructured notes. The framework uses a modular prompt template that can be adapted to new attributes without training labels, and for long-context attributes it relies on GPT-4-generated summaries of patient histories and clinical guidelines. The authors evaluate UMA on a suite of oncology attributes (PD-L1, performance status, treatment, response, progression, case finding, primary site, histology, and TNM staging) using real-world data from Providence Health and a held-out Johns Hopkins dataset. The central claim is that UMA with GPT-4o matches or exceeds attribute-specific supervised or heuristic baselines, achieves large gains on long-context staging tasks, generalizes better across institutions, and reduces cost and human effort by orders of magnitude.
Significance. If the claims hold, UMA would be a practically valuable contribution to clinical data structuring, potentially reducing the cost and time of registry curation and real-world evidence generation. The evaluation is substantial: the staging experiments use 2,918 Providence patients and 592 Johns Hopkins patients, with bootstrapped confidence intervals, external validation, ablations, and a cost analysis. The authors also explicitly release code, which is commendable. However, the paper's central interpretive claim that a single frontier model (e.g., GPT-4o) already possesses universal abstraction capability is not fully supported by the experimental design, because the decisive long-context results are obtained on GPT-4-summarized inputs and with a pipeline that includes an additional model (GPT-4) for preprocessing. The performance-status gold standard is also partially derived from an o3 model, which is in the same family as the evaluated models. These issues are fixable with additional experiments or a more calibrated claim, but they are load-bearing for the paper's headline message.
major comments (4)
- [Section 4.2 and Section 10.2, Figure S2] The claim that a 'single frontier model such as GPT-4o' achieves the long-context staging results is not directly supported by the experiments. For all nine long-context attributes, the extraction model never sees raw patient notes: the inputs are GPT-4-generated patient-history summaries and GPT-4-structured guideline summaries (Section 10.2, Figure S2). The ablation in Figure 3 shows that removing summarization hurts accuracy on fine-grained primary site, so this preprocessing materially contributes to the reported gains. To support the universal-abstraction claim, the paper should either run the full staging suite (or at least the attributes with the largest gains, such as pathologic T/N) on raw notes without the GPT-4 summarization step, or revise the abstract and Section 4.2 to attribute the gains to the UMA pipeline as a whole, explicitly including the GPT-4 preprocessing stage.
- [Section 10.1] The performance-status gold standard is a potential circularity: the initial annotations were made by one clinician, but model predictions from o3 were used to identify discrepancies, and a second clinician adjudicated those discrepancies. Because o3 belongs to the same model family as the evaluated models (GPT-4o, o1, o3), the resulting labels may be biased toward outputs that these models would produce. The paper should provide a sensitivity analysis excluding performance status from the overall average, or otherwise show that the o3-assisted adjudication did not systematically favor UMA.
- [Section 3 and Table S4] The paper does not compare UMA against a generic zero-shot LLM baseline (for example, GPT-4o prompted with the raw notes and a one-sentence attribute definition, without the UMA template or the summarization step). Without such a baseline, the contribution of the UMA template itself is not measured, and the 'matched or exceeded' claim conflates template design with underlying model capability. Adding this baseline would clarify whether the framework, rather than the model alone, drives the reported gains.
- [Abstract and Figure 2 caption] The abstract and Section 4.1 describe the baselines as 'state-of-the-art attribute-specific methods,' but for PD-L1, performance status, and treatment the baseline is a heuristic rule-based method rather than a trained model (Figure 2 caption, Table S4). The headline average improvement of roughly 2 points on Providence therefore partly reflects comparison against weak heuristics. The paper should either strengthen these baselines or qualify the claim to 'attribute-specific methods (supervised or heuristic).'
minor comments (5)
- [Table S1] In the Performance Status row, the descriptor 'performance status value' has example values 'CPS, TPS, expression,' which appear to be copy-pasted from the PD-L1 row; the correct values should be numeric or scale-based.
- [Figure 4] In the output format example, the second JSON key is again 'performance status measurement type' rather than 'performance status value,' which is presumably a typo in the prompt illustration.
- [Section 4.2] The sentence 'The second sub-figures in Section 10.2' is confusing; the referenced results are in Figure 2, and Section 10.2 is a methods section, not a figure location.
- [Section 4.1 and Discussion] The paper repeatedly states that the supervised baseline required 'over 10,000 hours of manual data curation,' but no citation or derivation for this estimate is provided; a reference or a calculation would make this claim more transparent.
- [Section 2.3] The text says patients often have '20+ notes spanning their clinical history,' yet Table S3 reports a median of 7 notes per patient for the long-context dataset; the example number appears inconsistent with the reported data.
Circularity Check
No significant circularity: UMA is evaluated on external registry and clinician labels, and the main gains are not fitted to those labels by construction. The GPT-4 summarization and o3-assisted adjudication are validity concerns, not circular derivation.
full rationale
The central quantitative claims rest on external ground truth: long-context staging labels come from cancer registries (Section 10.2) and short-context labels were manually curated by clinicians (Section 10.1). UMA is zero-shot; no attribute-specific parameter or prompt is fitted to the test labels, so the reported F1/accuracy gains are not forced by construction. The use of GPT-4 to summarize long patient histories and clinical guidelines is an explicit UMA preprocessing module; the summarization prompt (Figure S2) asks for clinically relevant content including staging information, so it may surface evidence, but the downstream model must still produce the normalized attribute value from the summary, and the summaries are not built from the gold labels. This is a system-comparison confound rather than a circular derivation. The performance-status gold standard was generated from a clinician's annotations with o3-model predictions used to trigger second-clinician adjudication (Section 10.1); this creates a same-family overlap between label construction and evaluated models, but because a human adjudicator makes the final decision, the labels are not by construction equal to any model output. The paper cites prior work by the same group for its supervised baseline and datasets, but those citations supply an independent baseline method and data rather than the conclusion, and no uniqueness theorem or ansatz is imported from that work. These concerns belong in correctness and validity assessment, not in the circularity score.
Assumptions & free parameters
assumptions (4)
- domain assumption The manual and cancer-registry ground-truth labels used for evaluation are accurate and representative.
- domain assumption The baselines labeled 'SOTA baseline' are representative of conventional attribute-specific medical abstraction methods.
- domain assumption The o3-assisted adjudication of performance status labels did not bias the gold standard toward LLM outputs.
- domain assumption GPT-4-generated summaries of clinical guidelines and patient histories preserve the information needed for staging decisions.
Cite this review
Pith. "Pith review of Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale." pith.science (2026). https://pith.science/paper/RTF44LEB
@misc{pith2026250200943,
author = {Pith},
title = {Pith review of: Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale},
year = {2026},
howpublished = {\url{https://pith.science/paper/RTF44LEB}},
note = {Machine review of arXiv:2502.00943}
}
read the original abstract
A significant fraction of real-world patient information resides in unstructured clinical text. Medical abstraction extracts and normalizes key structured attributes from free-text clinical notes, which is the prerequisite for a variety of important downstream applications, including registry curation, clinical trial operations, and real-world evidence generation. Prior medical abstraction methods typically resort to building attribute-specific models, each of which requires extensive manual effort such as rule creation or supervised label annotation for the individual attribute, thus limiting scalability. In this paper, we show that existing frontier models already possess the universal abstraction capability for scaling medical abstraction to a wide range of clinical attributes. We present UniMedAbstractor (UMA), a unifying framework for zero-shot medical abstraction with a modular, customizable prompt template and the selection of any frontier large language models. Given a new attribute for abstraction, users only need to conduct lightweight prompt adaptation in UMA to adjust the specification in natural languages. Compared to traditional methods, UMA eliminates the need for attribute-specific training labels or handcrafted rules, thus substantially reducing the development time and cost. We conducted a comprehensive evaluation of UMA in oncology using a wide range of marquee attributes representing the cancer patient journey. These include relatively simple attributes typically specified within a single clinical note (e.g. performance status), as well as complex attributes requiring sophisticated reasoning across multiple notes at various time points (e.g. tumor staging). Based on a single frontier model such as GPT-4o, UMA matched or even exceeded the performance of state-of-the-art attribute-specific methods, each of which was tailored to the individual attribute.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Predicting Early-Onset Colorectal Cancer with Large Language Models
Fine-tuned GPT-4o predicts early-onset colorectal cancer from EHR data 2-7 months before diagnosis with 73% sensitivity and 91% specificity, but with only 7.6% precision at 1% prevalence.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Kriti Bhattarai, Inez Y Oh, Jonathan Moran Sierra, Jonathan Tang, Philip RO Payne, Zach Abrams, and Albert M Lai. Leveraging gpt-4 for identifying cancer phenotypes in electronic health records: a performance comparison between gpt-4, gpt-3.5-turbo, flan-t5, llama-3-8b, and spacy’s rule-based and machine learning-based methods. JAMIA open, 7 0 (3): 0 ooae...
work page 2024
-
[3]
Classifying cancer pathology reports with hierarchical self-attention networks
Shang Gao, John X Qiu, Mohammed Alawad, Jacob D Hinkle, Noah Schaefferkoetter, Hong-Jun Yoon, Blair Christian, Paul A Fearn, Lynne Penberthy, Xiao-Cheng Wu, et al. Classifying cancer pathology reports with hierarchical self-attention networks. Artificial intelligence in medicine, 101: 0 101726, 2019
work page 2019
-
[4]
Limitations of transformers on clinical text classification
Shang Gao, Mohammed Alawad, M Todd Young, John Gounley, Noah Schaefferkoetter, Hong Jun Yoon, Xiao-Cheng Wu, Eric B Durbin, Jennifer Doherty, Antoinette Stroup, et al. Limitations of transformers on clinical text classification. IEEE journal of biomedical and health informatics, 25 0 (9): 0 3596--3607, 2021
work page 2021
-
[5]
Automating access to real-world evidence
Marie-Pier Gauthier, Jennifer H Law, Lisa W Le, Janice J N Li, Sajda Zahir, Sharon Nirmalakumar, Mike Sung, Christopher Pettengell, Steven Aviv, Ryan Chu, Adrian Sacher, Geoffrey Liu, Penelope Bradbury, Frances A Shepherd, and Natasha B Leighl. Automating access to real-world evidence. JTO Clin. Res. Rep., 3 0 (6): 0 100340, June 2022
work page 2022
-
[6]
Llms accelerate annotation for medical information extraction, 2023
Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, and Amir Feder. Llms accelerate annotation for medical information extraction, 2023
work page 2023
-
[7]
Javier Gonz \'a lez, Cliff Wong, Zelalem Gero, Jass Bagga, Risa Ueno, Isabel Chien, Eduard Oravkin, Emre Kiciman, Aditya Nori, Roshanthi Weerasinghe, et al. Trialscope: A unifying causal framework for scaling real-world evidence generation with biomedical language models. arXiv e-prints, pp.\ arXiv--2311, 2023
work page 2023
-
[8]
Improving large language models for clinical named entity recognition via prompt engineering
Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, and Hua Xu. Improving large language models for clinical named entity recognition via prompt engineering . Journal of the American Medical Informatics Association, pp.\ ocad259, 01 2024. ISSN 1527-974X. doi:10.10...
Show all 26 references
-
[9]
Generalizable and automated classification of tnm stage from pathology reports with external validation
Jenna Kefeli, Jacob Berkowitz, Jose M Acitores Cortina, Kevin K Tsang, and Nicholas P Tatonetti. Generalizable and automated classification of tnm stage from pathology reports with external validation. Nature Communications, 15 0 (1): 0 8916, 2024
2024
-
[10]
Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine
Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388 0 (13): 0 1233--1239, 2023
2023
-
[11]
Lost in the middle: How language models use long contexts
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024
2024
-
[12]
Cancer registrar workload and staffing study: Guidelines for hospital cancer registry programs
Jacqueline Miller, Laurie Hailer, and Susan Chapman. Cancer registrar workload and staffing study: Guidelines for hospital cancer registry programs. In 2024 Annual Research Meeting. AcademyHealth, 2024
2024
-
[13]
Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024. URL https://arxiv.org/abs/2410.05229
2024 arXiv
-
[14]
Salary considerations for cancer registrars: 2022
National Cancer Registrars Association . Salary considerations for cancer registrars: 2022. https://www.ncra-usa.org/Advocacy/Workforce/Salary-Considerations, 2022. Accessed June 11, 2025
2022
-
[15]
Revised recist guideline version 1.1: what oncologists want to know and what radiologists need to know
Mizuki Nishino, Jyothi P Jagannathan, Nikhil H Ramaiya, and Annick D Van den Abbeele. Revised recist guideline version 1.1: what oncologists want to know and what radiologists need to know. American Journal of Roentgenology, 195 0 (2): 0 281--289, 2010
2010
-
[16]
Can generalist foundation models outcompete special-purpose tuning? case study in medicine
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. Medicine, 84 0 (88.3): 0 77--3, 2023
2023
-
[17]
Toward structuring real-world data: Deep learning for extracting oncology information from clinical text with patient-level supervision
Sam Preston, Mu Wei, Rajesh Rao, Robert Tinn, Naoto Usuyama, Michael Lucas, Yu Gu, Roshanthi Weerasinghe, Soohee Lee, Brian Piening, Paul Tittel, Naveen Valluri, Tristan Naumann, Carlo Bifulco, and Hoifung Poon. Toward structuring real-world data: Deep learning for extracting ...
2023
-
[18]
Toward structuring real-world data: Deep learning for extracting oncology information from clinical text with patient-level supervision
Sam Preston, Mu Wei, Rajesh Rao, Robert Tinn, Naoto Usuyama, Michael Lucas, Yu Gu, Roshanthi Weerasinghe, Soohee Lee, Brian Piening, et al. Toward structuring real-world data: Deep learning for extracting oncology information from clinical text with patient-level supervision. ...
2023
-
[19]
Tnm classification
Ryan D Rosen and Amit Sapra. Tnm classification. In StatPearls [Internet]. StatPearls Publishing, 2023
2023
-
[20]
Opportunities and challenges in using real-world data for health care
Vivek A Rudrapatna, Atul J Butte, et al. Opportunities and challenges in using real-world data for health care. The Journal of clinical investigation, 130 0 (2): 0 565--574, 2020
2020
-
[21]
Estimating redundancy in clinical text
Thomas Searle, Zina Ibrahim, James Teo, and Richard Dobson. Estimating redundancy in clinical text. Journal of Biomedical Informatics, 124: 0 103938, December 2021. ISSN 1532-0464. doi:10.1016/j.jbi.2021.103938. URL http://dx.doi.org/10.1016/j.jbi.2021.103938
2021
-
[22]
Scaling clinical trial matching using large language models: A case study in oncology, 2023
Cliff Wong, Sheng Zhang, Yu Gu, Christine Moung, Jacob Abel, Naoto Usuyama, Roshanthi Weerasinghe, Brian Piening, Tristan Naumann, Carlo Bifulco, and Hoifung Poon. Scaling clinical trial matching using large language models: A case study in oncology, 2023
2023
-
[23]
Universalner: Targeted distillation from large language models for open named entity recognition, 2024
Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted distillation from large language models for open named entity recognition, 2024
2024
-
[24]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[25]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[26]
performance status measurement type
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
2016
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.