Pith. sign in

REVIEW 4 major objections 5 minor 23 references

The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that across 19,123 medical NLP studies, generative LLMs and traditional NLP occupy complementary niches: LLMs concentrate in open-ended tasks like education and summarization, while traditional NLP still leads structured…

desk verdict A useful 19k-paper map of medical NLP, undercut by an unvalidated keyword regex split and an overreaching 'advantages' claim. read the letter →

arxiv 2505.10261 v1 pith:RRN7KBOF submitted 2025-05-15 cs.CL cs.AI

Rui Yang , Huitao Li , Matthew Yu Heng Wong , Yuhe Ke , Xin Li , Kunyu Yu , Jingchi Liao , Jonathan Chong Kai Liew
show 7 more authors
Sabarinath Vinod Nair Jasmine Chiat Ling Ong Irene Li Douglas Teodoro Chuan Hong Daniel Shu Wei Ting Nan Liu
This is my paper · ORCID
classification cs.CLcs.AI
keywords generativelargelanguagemodelstraditionalNLPmedicalnaturalprocessingtopicmodelingBERTopicresearchlandscapeinformationextractioneducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper maps the current research landscape of medical natural language processing by analyzing 19,123 studies retrieved from four bibliographic databases. It argues that generative large language models and traditional NLP have complementary strengths, with LLM research concentrated in open-ended tasks such as medical education and text summarization, and traditional NLP still dominating structured information-extraction tasks such as named entity recognition and electronic health record mining. The authors combine keyword-based classification with neural topic modeling to measure how much of each topic's literature belongs to each technological paradigm. If the mapping is right, it tells researchers and clinicians which tool class to reach for, and where hybrid systems may be needed.

What carries the argument

The argument is carried by a topic-modeling pipeline: each study is embedded with a biomedical article encoder trained on PubMed query-article pairs, embeddings are reduced to four dimensions, clusters are found by density-based clustering, and topics are labeled and merged with expert review (with assistance from a large language model) to yield 26 topics. Keyword-based classification into generative LLM versus traditional NLP groups is applied to titles and abstracts first, then topic proportions are computed per group. The mechanism that produces the conclusion is the joint distribution of those two groupings: the same topic space, partitioned by paradigm, reveals where each paradigm's research attention concentrates.

What would settle it

Manually label a random sample of 300 abstracts from each group, compute the topic proportions from the corrected labels, and compare them to the paper's Figure 3; if the LLM group's share in open-ended topics drops sharply toward the traditional group's once passing mentions are removed, the central split is an artifact of the keyword classifier.

Watch

Extended reading notes

Core claim

The central discovery is an empirical division of labor: in a corpus of 19,123 studies, generative LLM papers (4,295) cluster in open-ended and generative use cases, while traditional NLP papers (14,828) keep the lead in structured extraction and analysis. Concretely, the topic 'Medical Education' has 72.23% of its studies in the LLM group, and 'Text Summarization' and 'Medical Image Analysis' are also LLM-heavy, whereas 'Electronic Health Records' (23.62%), 'Named Entity Recognition' (13.70%), and 'Semantic and Lexical Processing' (9.95%) dominate the traditional NLP share. The semantic-embedding visualization shows near-complete separation between the two groups along one dimension, with LLM studies more concentrated and traditional NLP more dispersed. The paper reads this as evidence that the two paradigms are complementary rather than mutually replacing: LLMs bring flexibility and generation, traditional methods bring controllability and precision.

Load-bearing premise

The whole comparison rests on classifying each paper by whether its title or abstract mentions generative-LLM keywords, so papers that merely compare against an LLM or mention one in passing are counted as LLM studies, and that noise could change the topic proportions.

Editorial extensions

If this is right

  • Medical NLP research appears to be splitting by task type, so future LLM work should target open-ended and generative functions while traditional NLP remains the default for structured extraction.
  • Hybrid systems that combine LLM generation with the precision of traditional extraction methods are a natural next step for electronic health record applications.
  • The near-complete semantic separation between the two groups suggests the field has not yet converged; as reasoning-capable LLMs mature, their share may extend into clinical reasoning and decision support.
  • For clinicians and educators, the concentration of LLM studies in medical education points to scalable simulation and self-assessment tools, provided ethical and privacy safeguards are in place.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper measures research activity, not measured performance; the word 'advantages' is an inference from where studies concentrate, not from benchmark results.
  • The near-perfect separation in the embedding space may be partly a mechanical effect of the keyword classification, since keywords such as 'LLM' appear in the abstracts that are embedded.
  • A cheap test of the mapping is to rerun the same pipeline on the same corpus with a manually labeled random sample of a few hundred abstracts; stable topic proportions would strengthen the claim, large shifts would call it into question.
  • Extending the search window past March 2025 could show whether the 72% share in medical education persists or migrates toward extraction tasks as LLMs gain instruction-following precision.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a bibliometric and topic-modeling analysis of 19,123 medical NLP studies retrieved from PubMed, Embase, Scopus, and Web of Science. The studies are classified into generative LLM and traditional NLP groups by keyword regex matching, embedded with the MedCPT Article Encoder, reduced with UMAP, clustered with BERTopic, and merged into final topic labels by medical experts. The authors report that generative LLM studies concentrate in topics such as Medical Education and Text Summarization, while traditional NLP studies concentrate in Electronic Health Records and Named Entity Recognition, and they conclude from these publication proportions that generative LLMs demonstrate advantages in open-ended tasks while traditional NLP dominates information extraction and analysis tasks.

Significance. If the descriptive claims were established, the paper would offer a useful map of where medical NLP research effort has been concentrated across two technological paradigms. The study has strengths: a large multi-database corpus, a domain-specific embedding model, an explicit keyword list in Supplementary D, and topic keyword tables in Supplementary B. However, the headline 'advantages' claim goes beyond the descriptive evidence because no performance or comparative evaluation data are reported, and the validity of the keyword-based binary split is not demonstrated. Reproducibility is also limited: no code repository is provided and the data availability statement is ambiguous. The paper's contribution is therefore better characterized as a descriptive landscape analysis than as an evaluation of task-specific advantages.

major comments (4)
  1. [Abstract] The abstract claims that 'generative LLMs demonstrate advantages in open-ended tasks, while traditional NLP dominates in information extraction and analysis tasks,' but the study reports no task-performance data or head-to-head evaluations. The cited evidence (e.g., Medical Education 72.23%, Text Summarization 19.95%, EHR 23.62%) consists of proportions of published studies per topic, and a higher publication share does not establish an advantage in task quality or accuracy. This inference is load-bearing for the central claim; the claim should be rephrased as a statement about research focus and publication activity, or supplemented with comparative performance evidence.
  2. [Methods: Data Processing / Supplementary D] The binary split into generative LLM versus traditional NLP is made by keyword regex matching in Supplementary D and is neither validated nor audited. The regex \b(palm)\b will match 'palmar' and other anatomical uses; \b(llama)\b will match llama-derived nanobodies; and \b(rag)\b will match the recombination-activating gene in biomedical text, so abstracts containing such terms plus a generic NLP phrase are counted as generative LLM studies. Conversely, the list omits patterns for GPT-3, GPT-4, OpenAI, o1, DeepSeek, Claude, and Mistral, and the search strategy in Supplementary C omits these terms except 'chatgpt', so many model-named generative LLM papers are likely absent from the corpus entirely. No precision, recall, manual audit, or inter-annotator agreement is reported. Because the counts 4,295 versus 14,828 and all topic proportions derive from this rule, the central comparison is not trustworthy until the classification rule is validated.
  3. [Methods: Topic Modeling / Supplementary B] The manuscript states that 40 initial topics were merged by medical experts into 26 topics, but Supplementary B lists 40 keyword sets with several duplicate topic labels (e.g., Medical Image Analysis appears as Topics 5, 9, and 34, and Mental Health & Psychology appears multiple times), so the mapping from initial to final topics is not fully documented. No inter-annotator agreement or reliability measure is reported for the expert merging. Since topic-level proportions are the basis for the main conclusions, the merging procedure needs to be transparent and reproducible.
  4. [Main Text, Figure 2 paragraph] The near-complete separation in Dimension 1 of the UMAP embedding is presented as evidence of disparity between the two groups, but this separation is expected because the grouping labels were assigned using keywords that appear in the same titles and abstracts used to compute the embeddings. This visualization should be presented as descriptive, with the caveat that it is partly a consequence of the classification rule, rather than as independent evidence for two distinct research paradigms.
minor comments (5)
  1. [Code Availability] The Code Availability section states that 'The data used in this study is available upon request,' which conflicts with the preceding Data Availability statement that the full data cannot be publicly shared; please clarify what artifacts (code, topic assignments, classification labels) are actually available and provide a code repository.
  2. [Supplementary Information A] Supplementary A is described as a PRISMA flow diagram, but the text provided contains only a narrative summary; please confirm that the actual flow diagram is included in the supplement.
  3. [References] Reference 19 is incomplete, consisting only of 'Website.' followed by a URL, and references 18 and 19 are non-archival web sources; please provide full citation details and access dates.
  4. [Main Text, Figure 3 cross-reference] The sentence 'Figure 2 shows the distribution ratio ...' appears to refer to Figure 3, which displays topic distributions; please correct the cross-reference.
  5. [Footnote 1] Footnote 1 contains the phrase 'a continuously period'; this should read 'a continuous period.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the bibliometric pipeline is external and the reported topic distributions are not forced by the paper's own classification rule.

full rationale

The paper's central claim is a descriptive comparison of topic proportions between papers matched by generative-LLM keywords (Supplementary D) and all remaining papers. The classification rule is a labeling choice, not a fitted parameter, and the topic proportions are computed from external embeddings (MedCPT Article Encoder), UMAP dimensionality reduction, HDBSCAN clustering, and BERTopic topic modeling. Nothing in the keyword list or the BERTopic procedure forces a generative-LLM paper to land in 'Text Summarization' or 'Medical Education' rather than 'Named Entity Recognition' or 'Electronic Health Records'; the observed distribution is an empirical outcome that could in principle have been different. The 'advantages' language in the abstract is an interpretive summary of those observed proportions, not a quantity derived by construction from the input keywords. The paper's self-citations (references 3, 9, 20) are background examples and are not load-bearing for the bibliometric analysis. The main legitimate concern about the keyword split is accuracy — possible false positives such as 'rag' matching recombination-activating gene or 'palm' matching palmar, and omitted model-name patterns — but that is a measurement-validity issue, not a circularity issue, because the conclusion does not reduce algebraically or definitionally to the grouping rule. No quoted step exhibits an equation that equals its input, no fitted parameter is renamed as a prediction, and no load-bearing result is imported solely from the authors' prior work. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

This paper makes no mathematical derivation and introduces no invisible entities. Its central claims depend on the assumptions above plus the unstated hyperparameters of the topic modeling pipeline. The fitted values are not physical parameters but analysis choices that shape the reported topic distribution.

free parameters (2)
  • UMAP embedding dimensionality = 4
    Hand-chosen reduction target; it affects cluster structure and the reported separation between generative and traditional NLP studies.
  • Number of BERTopic initial topics = 40
    Chosen or produced by the pipeline; the topic shares and merged categories depend on this count and on HDBSCAN settings.
assumptions (3)
  • domain assumption Keyword matching in titles and abstracts correctly classifies studies into generative LLM and traditional NLP groups.
    The entire comparison rests on the regular expressions in Supplementary D; papers that mention LLM or ChatGPT only as a comparison or in discussion are still assigned to the generative group.
  • domain assumption The 26 merged BERTopic topics correspond to meaningful medical task categories.
    The topic shares are interpreted as task distributions, but no external validation of topic coherence or label accuracy is reported beyond expert merging.
  • domain assumption MedCPT embeddings and UMAP projection preserve the semantic structure needed to separate the two groups.
    Near-complete separation in Dimension 1 is presented as evidence, but UMAP is a nonlinear projection whose global geometry is not guaranteed to preserve distances; cluster results can depend on hyperparameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine." pith.science (2026). https://pith.science/paper/RRN7KBOF

@misc{pith2026250510261,
  author       = {Pith},
  title        = {Pith review of: The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRN7KBOF}},
  note         = {Machine review of arXiv:2505.10261}
}
read the original abstract

Natural language processing (NLP) has been traditionally applied to medicine, and generative large language models (LLMs) have become prominent recently. However, the differences between them across different medical tasks remain underexplored. We analyzed 19,123 studies, finding that generative LLMs demonstrate advantages in open-ended tasks, while traditional NLP dominates in information extraction and analysis tasks. As these technologies advance, ethical use of them is essential to ensure their potential in medical applications.

Figures

Figures reproduced from arXiv: 2505.10261 by the authors.

Figure 1
Figure 1. Study workflow overview. We retrieved relevant studies from PubMed, Embase, Scopus, and Web of Science, and categorized them into generative LLM and traditional NLP groups based on keywords. We then performed topic modeling on all articles and merged similar topics. We identified a total of 44,609 studies from PubMed, Embase, Scopus, and Web of Science. After removing duplicates, 20,228 unique studies remained. We e… view at source ↗
Figure 2
Figure 2. Semantic embeddings for generative LLM and traditional NLP studies. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

  1. [1]

    Li, I. et al. Neural Natural Language Processing for unstructured data in electronic health records: A review. Comput. Sci. Rev. 46, 100511 (2022)

    work page 2022

  2. [2]

    M., Ohno-Machado, L

    Nadkarni, P. M., Ohno-Machado, L. & Chapman, W. W. Natural language processing: an introduction. J Am Med Inform Assoc 18, 544–551 (2011)

    work page 2011

  3. [3]

    Yang, R. et al. Ascle-A Python Natural Language Processing Toolkit for Medical Text Generation: Development and Evaluation Study. J Med Internet Res 26, e60601 (2024)

    work page 2024

  4. [4]

    Thirunavukarasu, A. J. et al. Large language models in medicine. Nat Med 29, 1930–1940 (2023)

    2023

  5. [5]

    Brown, T. B. et al. Language Models are Few-Shot Learners. arXiv [cs.CL] (2020) doi:10.48550/ARXIV.2005.14165

    arXiv doi:10.48550/arxiv.2005.14165 2020

  6. [6]

    GPT-4 Technical Report

    OpenAI et al. GPT-4 Technical Report. arXiv [cs.CL] (2023) doi:10.48550/ARXIV.2303.08774

    arXiv doi:10.48550/arxiv.2303.08774 2023

  7. [7]

    OpenAI o1 System Card

    OpenAI et al. OpenAI o1 System Card. arXiv [cs.AI] (2024) doi:10.48550/ARXIV.2412.16720

    arXiv doi:10.48550/arxiv.2412.16720 2024

  8. [8]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv [cs.CL] (2025) doi:10.48550/ARXIV.2501.12948

    arXiv doi:10.48550/arxiv.2501.12948 2025

Show all 23 references
  1. [9]

    Yang, R. et al. Large language models in health care: Development, 13 applications, and challenges. Health Care Sci 2, 255–263 (2023)

    2023

  2. [10]

    Ke, Y. et al. Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. J Med Internet Res 26, e59439 (2024)

    2024

  3. [11]

    Jin, Q. et al. MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics 39, (2023)

    2023

  4. [12]

    & Melville, J

    McInnes, L., Healy, J. & Melville, J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. (2018)

    2018

  5. [13]

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure

    Grootendorst, M. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. (2022) doi:10.48550/ARXIV.2203.05794

    2022 doi arXiv

  6. [14]

    Abd-Alrazaq, A. et al. Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions. JMIR Med Educ 9, e48291 (2023)

    2023

  7. [15]

    Tang, L. et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med 6, 158 (2023)

    2023

  8. [16]

    Sun, C. et al. Generative large language models trained for detecting errors in radiology reports. arXiv [cs.CL] (2025) doi:10.48550/ARXIV.2504.04336

    2025 doi arXiv

  9. [17]

    Hu, Y. et al. Improving large language models for clinical named entity recognition via prompt engineering. J Am Med Inform Assoc 31, 1812–1820 (2024). 14

    2024

  10. [18]

    Gemini 2.5: Our most intelligent AI model

    Kavukcuoglu, K. Gemini 2.5: Our most intelligent AI model. Google https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ (2025)

    2025

  11. [19]

    https://openai.com/index/o3-o4-mini-system-card/

    Website. https://openai.com/index/o3-o4-mini-system-card/

  12. [20]

    Yang, R. et al. KG-rank: Enhancing large language models for medical QA with knowledge graphs and ranking techniques. in Proceedings of the 23rd Workshop on Biomedical Natural Language Processing 155–166 (Association for Computational Linguistics, Stroudsburg, PA, USA, 2024)

    2024

  13. [21]

    Ning, Y. et al. Generative artificial intelligence and ethical considerations in health care: a scoping review and ethics checklist. Lancet Digit Health 6, e848–e856 (2024)

    2024

  14. [22]

    & Astels, S

    McInnes, L., Healy, J. & Astels, S. hdbscan: Hierarchical density based clustering. J. Open Source Softw. 2, 205 (2017)

    2017

  15. [23]

    natural language processing

    OpenAI et al. GPT-4o System Card. arXiv [cs.CL] (2024) doi:10.48550/ARXIV.2410.21276 15 Supplementary Information A: PRISMA flow diagram PRISMA flow diagram for identifying related studies. Our search retrieved 44,609 study records (n=8,145, 18.26% from PubMed; n=16,657, 37.34...

    2024 doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.