Pith. sign in

REVIEW 3 major objections 6 minor 33 references

Differentiating hype from practical applications of large language models in medicine -- a primer for healthcare professionals

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This primer argues that LLMs are statistical language parrots with no grasp of objective truth, so medical use of them must augment—not replace—human judgment.

desk verdict A solid, accessible primer whose practical message survives its own overreach; the categorical 'no understanding of truth' claim is contested but the human-in-the-loop advice stands. read the letter →

arxiv 2507.19567 v1 pith:F4LP23HJ submitted 2025-07-25 cs.CY cs.AI

Elisha D.O. Roberson This is my paper
classification cs.CYcs.AI
keywords largelanguagemodelsmedicineclinicaldecisionsupporthallucinationdataprivacyretrieval-augmentedgenerationAIsafetyhealthcareworkforce
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This primer for healthcare professionals tries to establish that large language models are non-thinking statistical language parrots: they learn patterns in text, not meaning, and they have no access to objective truth rooted in reality. Against the hype, the paper argues that two common assumptions are false: human-like responses do not imply critical thought, and LLM outputs do not carry any understanding of what is true. The stakes are concrete because medical use can harm real people, leak protected health information, or let institutions replace staff with systems that cannot be held liable. The practical conclusion is that LLMs should augment human judgment, never replace it, and every output that matters needs a human check.

What carries the argument

The machinery that carries the argument is the Transformer architecture understood as a statistical language model together with the octopus analogy that fixes its interpretation. The Transformer's multi-headed attention lets text be processed in parallel and weighted by context, which is why LLM responses sound human; the octopus analogy supplies the claim that this fluency is entirely form-based, with no knowledge map tying characters to real-world referents. On the applications side, the key mechanism is retrieval-augmented generation (RAG), which combines a trained model with dynamic external sources and human oversight so that outputs are grounded in retrievable data rather than pure pattern completion.

What would settle it

One concrete way to test the central claim is a controlled benchmark in which a purely text-trained model, with no retrieval or human feedback, is asked novel questions about physical or causal facts that are not stated in any training text; if it answers accurately by genuine inference, the claim that meaning cannot come from form alone would be challenged. A simpler observational check is to audit a production medical LLM's reference lists: consistent correct DOIs to real papers would undercut the claim that hallucination is architecturally inevitable.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM is a statistical model trained to produce plausible next tokens, so its fluent answers are pattern completions rather than reasoned statements about the world. It adopts the octopus thought experiment as the working model: a system that masters the statistical relationship between messages can imitate a correspondent perfectly while remaining ignorant of bears, weapons, and every other fact the messages refer to. From this, the paper concludes that hallucinations are not occasional bugs but an expected behavior of the architecture, that confidential data pasted into a model can be extracted by other users, and that biased or false patterns in the training corpus will flow through to clinical answers. The constructive half of the paper is that LLMs remain useful as augmentation tools when paired with retrieval-augmented generation, domain-specific systems, and human oversight.

Load-bearing premise

The argument's load-bearing premise is the form hypothesis: a system trained only on language form, without grounding in the world, cannot acquire meaning or truth; if grounding can arise from text alone, the claimed architectural limits are overstated.

Editorial extensions

If this is right

  • Clinical workflows that rely on LLM summaries must include a human verification step, because confident responses can mix correct and fabricated content within a single reference or answer.
  • Pasting patient data, unpublished manuscripts, or confidential documents into public LLM interfaces risks exposing protected information to other users through training-data extraction.
  • Institutions considering LLM-based staff replacement should expect confidentiality breaches, biased outputs, and liability to remain with the human organization, since no machine can be held responsible.
  • Retrieval-augmented generation and domain-specific biomedical models are the safer deployment pattern, but they reduce rather than eliminate hallucinations.
  • Automated monitoring and scheduling systems are promising because their outputs are testable or constrained, whereas LLM recommendations in diagnosis and treatment need stricter human oversight.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The author leaves implicit that if hallucination is truly architectural, then regulatory frameworks for medical AI should require disclosure of LLM involvement and traceable sources for every generated statement, not just better benchmarks.
  • The same reasoning extends to scientific publishing: if LLM-assisted peer review and paper mills contaminate the training corpus, future models will absorb the contamination, creating a feedback loop that degrades biomedical knowledge.
  • A testable extension would be comparing RAG-grounded LLM summaries against plain LLM summaries on a realistic clinical task, measuring both factual accuracy and clinician time saved; the paper's framework predicts RAG plus human oversight wins.
  • The criteria drift evidence suggests that user perception of LLM quality changes during a session, so satisfaction surveys likely overstate real-world reliability; objective accuracy checks should be built into deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript is a primer for healthcare professionals on large language models (LLMs) in medicine. It explains foundational concepts (neural networks, transformers, tokens, parameters), argues that LLMs are "non-thinking statistical language parrots" with no understanding of objective truth, reviews risks (confidentiality, hallucinations, data poisoning, training bias), and surveys practical applications such as non-programmatic interfaces, unstructured-data parsing, embeddings and retrieval-augmented generation, natural-language summaries, document/code skeletons, constrained-solution problems, and continuous monitoring. It concludes that LLM deficiencies may be architectural and possibly unsolvable, and that medical deployment should augment rather than replace human judgment.

Significance. The paper is readable, accessible, and covers a useful breadth of practical issues with concrete examples and current references; the simple blood-pressure neural-network illustration and the hallucinated-reference example are pedagogically effective. The broad recommendation that LLMs should augment, not replace, human clinical judgment is well motivated by multiple independent considerations, including confidentiality, accountability, and the cited meta-analysis showing roughly 56% overall accuracy. However, the paper's central categorical claims about LLM understanding and the inevitability of hallucination rest on a contested philosophical premise and an overgeneralized formal result; these need qualification. The contribution is synthesis and perspective rather than original evidence; there are no machine-checked proofs or fitted parameters, and the illustrative LLM query is anecdotal.

major comments (3)
  1. [pp.5–6 and p.10 (Conclusions)] The paper states as established fact that LLMs have "no understanding of objective truth that is based in reality" and that "both of these assumptions are false," and it suggests the deficiencies "may, in fact, be a limitation of the architecture of LLMs that will never be solved." The supporting argument is the Bender–Koller octopus analogy (ref. 5), which is a contested philosophical thesis about the impossibility of acquiring meaning from form alone, not a proven result. The manuscript does not acknowledge that this position is actively debated, nor does it engage with evidence that large text corpora can support latent world-model induction. Since these categorical statements appear in the abstract and conclusions and underpin the "human in the loop" recommendation, the primer should be revised to present the claim as a prominent safety perspective rather than a proven fact, and to state what evidence would count against it.
  2. [Hallucinations section, p.7] The sentence "removing hallucinations completely may be impossible due to the internal LLM architecture" is supported only by Xu et al. (ref. 12). That result is a theorem about a specific formal definition of hallucination under standard autoregressive sampling; it does not cover retrieval-augmented generation or constrained decoding, approaches the paper itself recommends on pp.8–9. As written, the sentence overgeneralizes the theorem and provides the basis for the stronger "never be solved" conclusion. The manuscript should specify the formal setting of the impossibility result and clarify that practical mitigation, including RAG and constrained decoding, can reduce but not eliminate hallucination risk.
  3. [Hallucinations section, pp.6–7 and Fig. 2] The "part true and part hallucination" example is a single illustrative query to ChatGPT 3.5 (Fig. 2). It is used to support the general claim that LLM outputs are intrinsically unreliable, but a single anecdote cannot bear that weight. The paper should explicitly label this as an illustrative anecdote, distinguish it from systematic evaluations of hallucination rates in biomedical settings, and avoid using it as evidence for the absolute impossibility claim. This is load-bearing because the example is one of the few concrete pieces of evidence offered for the paper's central position.
minor comments (6)
  1. [Fig. 2 legend, p.7] The legend contains a typo: "ChapGPT 3.5" should be "ChatGPT 3.5."
  2. [Confidentiality section, p.6] The statement that "the queries submitted to an LLM often become tokens in the model" is imprecise: input prompts are not necessarily incorporated into the model's training corpus; they may be logged, stored, or used for fine-tuning depending on the deployment. The privacy risk is real, but the wording gives readers an inaccurate mental model of how training data are formed.
  3. [Applications section, pp.8–9] The word "corpuses" appears twice; the standard plural is "corpora."
  4. [Conclusions, p.10] The sentence "no machine can be held liable for its actions" is too absolute; legal liability for AI-related harm currently attaches to deployers, manufacturers, or operators in many jurisdictions. Suggest rephrasing to note that a machine itself cannot be held liable, while human or organizational liability may still apply.
  5. [Bibliography, pp.11–13] The reference formatting is inconsistent: some entries use plain URLs, some use DOIs, and some preprint citations lack version or access dates. A uniform style would improve the primer's usability.
  6. [What is a large language model?, p.4] The phrase "A key advantage of machine learning: identifying patterns in training even if the human users don't understand the rules underlying the data" should read "in training data" for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a review-style primer whose claims rest on external literature and illustrative examples, not on a derivation chain that reduces to its own inputs.

full rationale

The paper contains no fitted parameters, no equations that define one claimed result in terms of another, and no self-citation chain used as the load-bearing justification. The central claim that LLMs are 'non-thinking statistical language parrots' with 'no understanding of objective truth that is based in reality' is supported by the Bender and Koller octopus analogy and by citations to external, independently published work. That cited work includes the external 'Hallucination is Inevitable' theorem (Xu et al.) and empirical studies of training-data extraction, data poisoning, bias, and accuracy. The single illustrative ChatGPT query about references is used as a demonstration of hallucination, not as an input fitted to produce the paper's conclusions. Even if the Bender-Koller form hypothesis is contested, or if the cited theorem is narrower than the paper's absolutist framing, those are concerns about evidential strength and correctness risk, not circularity. The paper does not define its conclusion into its premises, does not rename a known pattern as a new result, and does not rely on the author's own prior work. Accordingly, no circular step can be exhibited, and the honest finding is no significant circularity with a score of 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters and no invented entities. Its conclusion depends on two adopted assumptions from the cited literature: the form-grounding hypothesis and the architectural inevitability of hallucination. Both are domain assumptions, not standard mathematical axioms.

assumptions (2)
  • domain assumption A model trained only on linguistic form cannot acquire semantic understanding or ground truth.
    The paper's octopus analogy adopts this form hypothesis from Bender and Koller (ref 5) and uses it as the foundation for the claim that LLMs are non-thinking parrots.
  • domain assumption Hallucination is an unavoidable property of current LLM architectures.
    The paper cites Xu et al. (ref 12) for inevitability and relies on it to argue that no training-scale solution will fix LLM reliability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentiating hype from practical applications of large language models in medicine -- a primer for healthcare professionals." pith.science (2026). https://pith.science/paper/F4LP23HJ

@misc{pith2026250719567,
  author       = {Pith},
  title        = {Pith review of: Differentiating hype from practical applications of large language models in medicine -- a primer for healthcare professionals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4LP23HJ}},
  note         = {Machine review of arXiv:2507.19567}
}
read the original abstract

The medical ecosystem consists of the training of new clinicians and researchers, the practice of clinical medicine, and areas of adjacent research. There are many aspects of these domains that could benefit from the application of task automation and programmatic assistance. Machine learning and artificial intelligence techniques, including large language models (LLMs), have been promised to deliver on healthcare innovation, improving care speed and accuracy, and reducing the burden on staff for manual interventions. However, LLMs have no understanding of objective truth that is based in reality. They also represent real risks to the disclosure of protected information when used by clinicians and researchers. The use of AI in medicine in general, and the deployment of LLMs in particular, therefore requires careful consideration and thoughtful application to reap the benefits of these technologies while avoiding the dangers in each context.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

  1. [1]

    Artificial Intelligence: A Modern Approach

    Stuart Russel, Peter Norvig. Artificial Intelligence: A Modern Approach. Fourth Edition. Pearson

    work page

  2. [2]

    Predicting increased blood pressure using Machine Learning

    Hudson Golino. Women’s dataset from the “Predicting increased blood pressure using Machine Learning” paper. doi:https://doi.org/10.6084/m9.figshare.845664.v1

    work page doi:10.6084/m9.figshare.845664.v1

  3. [3]

    Attention Is All You Need

    Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need. Published online August 1, 2023. doi:10.48550/arXiv.1706.03762

    arXiv doi:10.48550/arxiv.1706.03762 2023

  4. [4]

    A. M. Turing. Computing machinery and intelligence. Mind. 49:433-460

    work page

  5. [5]

    Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

    Bender EM, Koller A. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In: Jurafsky D, Chai J, Schluter N, Tetreault J, eds. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2020:5185-5198. doi:10.18653/v1/2020.acl-main.463

    doi:10.18653/v1/2020.acl-main.463 2020

  6. [6]

    Scalable Extraction of Training Data from (Production) Language Models

    Nasr M, Carlini N, Hayase J, et al. Scalable Extraction of Training Data from (Production) Language Models. Published online November 28, 2023. doi:10.48550/arXiv.2311.17035

    arXiv doi:10.48550/arxiv.2311.17035 2023

  7. [7]

    Stealing Part of a Production Language Model

    Carlini N, Paleka D, Dvijotham KD, et al. Stealing Part of a Production Language Model. Published online March 11, 2024. doi:10.48550/arXiv.2403.06634

    arXiv doi:10.48550/arxiv.2403.06634 2024

  8. [8]

    Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack

    Russinovich M, Salem A, Eldan R. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. Published online April 2, 2024. doi:10.48550/arXiv.2404.01833

    arXiv doi:10.48550/arxiv.2404.01833 2024

Show all 33 references
  1. [9]

    Avianca, Inc.(United States District Court, Southern District of New York 2023)

    Mata v. Avianca, Inc.(United States District Court, Southern District of New York 2023). https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd .575368.54.0_3.pdf

    2023

  2. [10]

    Artificial Hallucinations in ChatGPT: Implications in Scientific Writing

    Alkaissi H, McFarlane SI. Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus. 2023;15(2):e35179. doi:10.7759/cureus.35179

    2023 doi

  3. [11]

    GeneGPT: augmenting large language models with domain tools for improved access to biomedical information

    Jin Q, Yang Y, Chen Q, Lu Z. GeneGPT: augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics. 2024;40(2):btae075. doi:10.1093/bioinformatics/btae075

    2024 doi

  4. [12]

    Hallucination is Inevitable: An Innate Limitation of Large Language Models

    Xu Z, Jain S, Kankanhalli M. Hallucination is Inevitable: An Innate Limitation of Large Language Models. Published online January 22, 2024. doi:10.48550/arXiv.2401.11817 Page 12 of 13 Roberson 2025 – LLMs in medicine

    2024 doi arXiv

  5. [13]

    Exposing Vulnerabilities in Clinical LLMs Through Data Poisoning Attacks: Case Study in Breast Cancer

    Das A, Tariq A, Batalini F, Dhara B, Banerjee I. Exposing Vulnerabilities in Clinical LLMs Through Data Poisoning Attacks: Case Study in Breast Cancer. Published online March 21, 2024:2024.03.20.24304627. doi:10.1101/2024.03.20.24304627

    2024 doi

  6. [14]

    Poisoning scientific knowledge using large language models

    Yang J, Xu H, Mirzoyan S, et al. Poisoning scientific knowledge using large language models. Published online November 10, 2023:2023.11.06.565928. doi:10.1101/2023.11.06.565928

    2023 doi

  7. [15]

    Large language models propagate race-based medicine

    Omiye JA, Lester JC, Spichak S, Rotemberg V, Daneshjou R. Large language models propagate race-based medicine. NPJ Digit Med. 2023;6(1):195. doi:10.1038/s41746-023-00939-z

    2023 doi

  8. [16]

    A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records

    Chiang CC, Luo M, Dumkrieger G, et al. A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records. Headache. Published online March 25, 2024. doi:10.1111/head.14702

    2024 doi

  9. [17]

    Large language models to identify social determinants of health in electronic health records

    Guevara M, Chen S, Thomas S, et al. Large language models to identify social determinants of health in electronic health records. NPJ Digit Med. 2024;7(1):6. doi:10.1038/s41746-023-00970-0

    2024 doi

  10. [18]

    Retrieval-augmented generation for knowledge- intensive NLP tasks

    Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge- intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS ’20. Curran Associates Inc.; 2020:9459- 9474

    2020

  11. [19]

    Unveiling differential adverse event profiles in vaccines via LLM text embeddings and ontology semantic analysis

    Wang Z, Li X, Zheng J, He Y. Unveiling differential adverse event profiles in vaccines via LLM text embeddings and ontology semantic analysis. J Biomed Semantics. 2025;16(1):10. doi:10.1186/s13326-025-00331-8

    2025 doi

  12. [20]

    Dual retrieving and ranking medical large language model with retrieval augmented generation

    Yang Q, Zuo H, Su R, et al. Dual retrieving and ranking medical large language model with retrieval augmented generation. Sci Rep. 2025;15:18062. doi:10.1038/s41598-025-00724-w

    2025 doi

  13. [21]

    Empowering PET imaging reporting with retrieval-augmented large language models and reading reports database: a pilot single center study

    Choi H, Lee D, Kang Y koo, Suh M. Empowering PET imaging reporting with retrieval-augmented large language models and reading reports database: a pilot single center study. Eur J Nucl Med Mol Imaging. 2025;52(7):2452-2462. doi:10.1007/s00259-025-07101-9

    2025 doi

  14. [22]

    GICL: A Cross-Modal Drug Property Prediction Framework Based on Knowledge Enhancement of Large Language Models

    Li N, Qiao J, Gao F, et al. GICL: A Cross-Modal Drug Property Prediction Framework Based on Knowledge Enhancement of Large Language Models. J Chem Inf Model. 2025;65(11):5518-5527. doi:10.1021/acs.jcim.5c00895

    2025 doi

  15. [23]

    CDEMapper: enhancing National Institutes of Health common data element use with large language models

    Wang Y, Huang J, He H, et al. CDEMapper: enhancing National Institutes of Health common data element use with large language models. J Am Med Inform Assoc. 2025;32(7):1130-1139. doi:10.1093/jamia/ocaf064 Page 13 of 13 Roberson 2025 – LLMs in medicine

    2025 doi

  16. [24]

    ChatGPT usage in the Reactome curation process

    Krishna Kumar Tiwari, Lisa Matthews, Bruce May, et al. ChatGPT usage in the Reactome curation process. bioRxiv. Published online January 1, 2023:2023.11.08.566195. doi:10.1101/2023.11.08.566195

    2023 doi

  17. [25]

    The application of ChatGPT in healthcare progress notes: A commentary from a clinical and research perspective

    Nguyen J, Pepping CA. The application of ChatGPT in healthcare progress notes: A commentary from a clinical and research perspective. Clin Transl Med. 2023;13(7):e1324. doi:10.1002/ctm2.1324

    2023 doi

  18. [26]

    Semantic search using protein large language models detects class II microcins in bacterial genomes

    Kulikova AV, Parker JK, Davies BW, Wilke CO. Semantic search using protein large language models detects class II microcins in bacterial genomes. Published online November 15, 2023:2023.11.15.567263. doi:10.1101/2023.11.15.567263

    2023 doi

  19. [27]

    Deep embedding and alignment of protein sequences

    Llinares-López F, Berthet Q, Blondel M, Teboul O, Vert JP. Deep embedding and alignment of protein sequences. Nat Methods. 2023;20(1):104-111. doi:10.1038/s41592-022-01700-2

    2023 doi

  20. [28]

    Automated design of protein-binding riboswitches for sensing human biomarkers in a cell-free expression system

    Vezeau GE, Gadila LR, Salis HM. Automated design of protein-binding riboswitches for sensing human biomarkers in a cell-free expression system. Nat Commun. 2023;14(1):2416. doi:10.1038/s41467-023-38098-0

    2023 doi

  21. [29]

    Predicting Splicing from Primary Sequence with Deep Learning

    Jaganathan K, Kyriazopoulou Panagiotopoulou S, McRae JF, et al. Predicting Splicing from Primary Sequence with Deep Learning. Cell. 2019;176(3):535- 548.e24. doi:10.1016/j.cell.2018.12.015

    2019 doi

  22. [30]

    RUDEUS, a machine learning classification system to study DNA-Binding proteins

    Medina-Ortiz D, Cabas-Mora G, Moya-Barría I, Soto-Garcia N, Uribe-Paredes R. RUDEUS, a machine learning classification system to study DNA-Binding proteins. Published online February 21, 2024:2024.02.19.580825. doi:10.1101/2024.02.19.580825

    2024 doi

  23. [31]

    TemBERTure: Advancing protein thermostability prediction with Deep Learning and attention mechanisms

    Rodella C, Lazaridi S, Lemmin T. TemBERTure: Advancing protein thermostability prediction with Deep Learning and attention mechanisms. Published online March 31, 2024:2024.03.28.587204. doi:10.1101/2024.03.28.587204

    2024 doi

  24. [32]

    Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

    Shankar S, Zamfirescu-Pereira JD, Hartmann B, Parameswaran A, Arawjo I. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. In: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. UIST ’24. A...

    2024

  25. [33]

    Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis

    Wei Q, Yao Z, Cui Y, Wei B, Jin Z, Xu X. Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis. J Biomed Inform. 2024;151:104620. doi:10.1016/j.jbi.2024.104620

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.