OpenAI ChatGPT interprets Radiological Images: GPT-4 as a Medical Doctor for a Fast Check-Up

Enis Karaarslan; Omer Aydin

REVIEW 4 major objections 6 minor 10 references

OpenAI ChatGPT interprets Radiological Images: GPT-4 as a Medical Doctor for a Fast Check-Up

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4o correctly identified only one of four chest X-rays, so the paper concludes it cannot yet make final diagnoses on its own.

desk verdict A transparent but tiny four-image case study; the 25% figure depends on an exact-etiology rubric and no human baseline, so the conclusions outrun the evidence. read the letter →

arxiv 2501.06269 v2 pith:2QTWNKJO submitted 2025-01-09 cs.CV

Omer Aydin , Enis Karaarslan This is my paper · ORCID

classification cs.CV

keywords OpenAIChatGPTGPT-4ArtificialIntelligenceDecisionSupportSystemMedicalDoctorchestX-rayinterpretationcovid-chestxray-dataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether OpenAI's GPT-4o can interpret chest X-rays well enough to replace a medical doctor or to serve as a decision-support tool. The authors test four labeled lung images from a public COVID-era dataset, presenting them both as a four-panel composite and individually. GPT-4o correctly identified only the COVID-19 film in the composite (a 25% success rate) and, when shown individual images, correctly recognized the healthy case while misreading bacterial pneumonia as a clavicle fracture, viral pneumonia as bacterial, and COVID-19 as generalized pneumonia. The authors conclude that ChatGPT is not sufficiently accurate to make a final diagnosis in its current form, but that its structured, criterion-based responses could assist doctors and clinicians.

What carries the argument

The carrying mechanism is a prompted image-interpretation protocol: GPT-4o receives chest X-rays as image inputs, is instructed to act as a radiologist, and is asked to classify each film as healthy or sick and, if sick, by cause (bacteria, virus, COVID-19, or other). The model's outputs are compared against the ground-truth labels of the covid-chestxray-dataset, with the same images tested in composite and individually to probe consistency.

What would settle it

Take the same four chest X-rays, have two radiologists independently confirm the labels, then prompt GPT-4o again with the four panels in a randomized order and at full resolution; if the model gets more than one of four correct, the paper's 25% success rate does not generalize.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a state-of-the-art general-purpose multimodal chatbot, prompted to act as a radiologist, produces fluent and anatomically structured readouts yet fails on three of four chest X-rays when judged against the dataset's labels. The model's only unambiguous success was identifying a healthy film as normal; it missed bacterial pneumonia, confused viral pneumonia with bacterial infection, and did not name COVID-19 as the likely cause of the fourth film even though it flagged pneumonia. The paper therefore establishes a negative result about standalone use and a weakly evidenced positive result about assistance: the model can enumerate radiographic criteria and recommend next steps, but its diagnostic accuracy is too low for autonomous deployment.

Load-bearing premise

The whole accuracy count depends on the four images' labels being correct and the composite figure being laid out in the assumed order, since neither was independently verified by a radiologist.

Editorial extensions

If this is right

General-purpose multimodal LLMs should not be deployed as standalone chest X-ray readers in their current form.
Their structured radiology-style reports could still serve as a triage or educational aid, but every output needs clinician verification.
Diagnostic performance is sensitive to how images are presented, so evaluations must standardize layout and resolution.
Specialized training on infectious etiologies and integration of clinical data are prerequisite steps before such tools can be relied upon.
The model's correct read of the healthy film hints that ruling out obvious disease may be a more attainable near-term use than pinpointing etiology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The measured 25% success rate could be an artifact of the composite layout: if GPT-4o misread which panel was labeled 1, 2, 3, or 4, its diagnoses might have been correct for the wrong images. A repeat with clearly labeled, full-resolution panels would separate spatial reasoning from image interpretation.
The ground-truth labels are taken from the dataset's filenames without independent radiologist re-reads; a label error in any of the four films would change the success count. Re-verifying the four images is a cheap way to test the paper's core number.
A larger follow-up with dozens of films, multiple prompt phrasings, and statistical confidence intervals would be needed before using these results to decide real-world deployment.
The paper's optimism about assistance rests on process quality (fluent, structured reports), not on measured accuracy; unless assistance is evaluated separately, the claim that the model 'can assist' remains unproven.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a four-image case study of GPT-4o reading chest X-rays from the covid-chestxray-dataset. What the paper does well is show its work: every prompt and full response is quoted. That transparency is real. It is also honest about the tiny sample and says more experiments are needed.

What is new is almost nothing. Prior evaluations of GPT-4 on chest X-rays already exist, and the paper cites Lecler et al. The only addition is a composite-image prompt (four X-rays in one figure) plus individual prompts, with n=4 and no repeated sampling.

The soft spots are bigger than the positive content. The 25% success rate is computed against an exact-etiology rubric (bacterial vs viral vs COVID vs healthy). Under a clinically more standard sick/healthy rubric, the model read three of four images as showing pneumonia or disease; only the bacterial case was read as trauma. Chest X-ray has limited ability to separate bacterial, viral, and COVID-19 pneumonia on a single image, so requiring exact etiology may not be a fair accuracy criterion. Without a radiologist baseline on these same four images under the same rubric, the central negative claim is not established. The ground-truth labels come from dataset metadata with no independent verification. The composite trial also depends on the assumption that ChatGPT mapped the numbered labels to the correct positions, which the authors themselves flag. And the "can assist doctors" claim is untested: no clinician ever looked at the outputs.

I'd still say the paper is a serious attempt; it doesn't hide its failures. But the evidence is too thin and the design too cursory to support the abstract's conclusions. A peer reviewer would need major revisions: repeated trials, a human-reader baseline, label verification, and a clinician assessment of the assist value.

My recommendation: I wouldn't send this to peer review as a full research paper. It is more of a preprint-level case note. It might be worth a reading group discussion about evaluation rubrics, but it shouldn't be cited as evidence for or against GPT-4 performance.

Referee Report

4 major / 6 minor

Summary. The paper reports a small case study in which ChatGPT-4o was asked to interpret four chest X-ray images from the covid-chestxray-dataset. The authors first present a composite test with all four images and then individual tests, comparing the model's free-text diagnoses with dataset labels. They report a 25% success rate in the composite test, note that the model correctly identified only the COVID-19 case, and conclude that ChatGPT is not sufficiently accurate for standalone diagnosis but could assist clinicians. The paper includes verbatim prompts and responses, a discussion of each case, and recommendations for future work.

Significance. If the central claim were established, the paper would provide a useful cautionary data point about general-purpose vision-language models in radiology. The authors are transparent about their prompts and record the raw outputs, and they use a public dataset, which supports reproducibility. However, the study is only a four-image, single-trial case report without a human-reader baseline or clinician evaluation, and the exact-etiology rubric conflates triage ability with etiologic specificity. The assistive claim is not measured at all. These gaps make the current evidence too weak to support the abstract's conclusions, although the descriptive material is a reasonable starting point.

major comments (4)

[Abstract and Implementation (composite test)] The central negative claim that ChatGPT is 'not sufficient and accurate' is derived from an exact-etiology scoring of four images (25% in the composite test). However, the verbatim responses in the Implementation section show that the model correctly distinguished sick from healthy in three of the four individual cases: the healthy image was read as normal, the viral pneumonia image was read as pneumonia (wrong cause), and the COVID-19 image was read as pneumonia (not specifically COVID-19); only the bacterial case was missed entirely as a clavicle fracture. Because chest radiography has limited ability to differentiate bacterial, viral, and COVID-19 pneumonia from a single image, the exact-etiology rubric may not be an appropriate accuracy criterion. Without a radiologist baseline on the same four images under the same rubric, the reported 25% cannot support the conclusion that the model is not sufficiently accurate for triage or for identifying the presence of disease.
[Conclusion] The paper's positive claim that ChatGPT 'can provide interpretations that can assist medical doctors or clinicians' is untested. No clinician or radiologist reviewed the model's outputs for usefulness, correctness of the reasoning, or safety in a decision-support workflow. The Conclusion's statement that the verbatim responses provide 'strong evidence' for an active role in medical image processing is therefore unsupported by the reported methods, which only compare outputs with dataset labels.
[Implementation] The accuracy result is conditional on two unverified assumptions. First, the ground-truth labels are taken from the covid-chestxray-dataset metadata without independent radiologist confirmation that each image is correctly labeled (Data and Method, Table 1). Second, the composite test requires that ChatGPT interpreted the numbered positions in Figure 1 in the intended order; the paper itself states that the 25% success rate applies only 'If ChatGPT did not examine the wrong images due to a misunderstanding of the locations.' If either assumption fails, the reported success rate and the per-case conclusions do not hold.
[Data and Method] The study consists of a single trial per image and per composite presentation with no repeated sampling of the stochastic model. There is no statistical analysis, confidence interval, or human-reader comparison, so the reported accuracy estimates cannot be separated from sampling variation. Adding multiple independent runs, a small set of experienced readers, and a pre-specified scoring rubric would be needed to make the central claim robust.

minor comments (6)

[Data and Method (Table 1)] The in-text citation 'Chon et al. (2020)' should be 'Cohen et al. (2020)'; the reference list correctly cites Cohen et al., so the in-text name is inconsistent.
[References] There is a typo in the reference for Sechopoulos et al.: 'digital übreast tomosynthesis' should be 'digital breast tomosynthesis.'
[General] The paper uses inconsistent terminology for the virus ('virus' vs 'COVID-19' vs 'Covid'); standardizing to 'COVID-19' throughout would improve readability.
[References and Related Work] Several listed references (e.g., Lecler et al., 2023) are not cited in the body, and the related-work section would benefit from an explicit discussion of existing vision-language model evaluations in chest X-ray interpretation.
[Figures] Some figures (e.g., Figure 2 and Figures 3–6) are not explicitly called out in the text at the point where their analysis is discussed; adding in-text references would help the reader map the responses to the images.
[Conclusion] The claim 'The success rate is very low' should be quantified (e.g., 1/4 in the composite test) so that the reader can assess the magnitude without re-reading the Implementation section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is an empirical benchmark against an external public dataset, with no fitted parameters, self-cited derivation chain, or definitional reduction forcing the reported result.

full rationale

The paper's central claim is that ChatGPT-4o is not sufficiently accurate for standalone chest X-ray diagnosis but may assist clinicians. This claim is derived from direct, recorded ChatGPT-4o responses to four chest X-ray images from the public covid-chestxray-dataset, compared against the dataset's known labels. There are no equations, no fitted parameters, and no internal model whose output is defined in terms of the target conclusion. The authors' own prior works are cited only for background topics such as adversarial robustness and privacy, not as load-bearing justification for the diagnostic accuracy finding. The acknowledged caveats, such as the possibility that ChatGPT misread image locations in the composite trial, weaken the strength of the 25% success-rate claim as an evidentiary matter, but they do not make the derivation circular. The ground-truth label assumption is a data-quality or external-validity concern, not a circularity concern. Accordingly, no circularity step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No equations, fitted numbers, or hand-tuned constants appear; the 25% success rate is a summary of four binary judgments, not a parameter. The axioms are two data-quality assumptions and one representativeness assumption, all of which the paper takes for granted. No new entities, forces, or mediators are introduced; future-directions topics such as zero-knowledge proofs and blockchain are not central to the claim.

assumptions (3)

domain assumption The ground-truth labels in the covid-chestxray-dataset (Cohen et al., 2020) are clinically accurate for the four selected images.
The paper scores ChatGPT-4o against these labels (Data and Method, Table 1) without independent radiologist verification; an incorrect label would invalidate the accuracy numbers.
domain assumption GPT-4o's natural-language responses to a single prompt are a valid measure of its visual diagnostic ability.
The study treats each quoted response as the model's diagnosis and does not repeat prompts, sample temperature, or check whether the model actually attended to the intended image.
domain assumption The four selected images are representative of the categories bacterial, healthy, viral, and COVID-19.
The images were chosen from a larger dataset with no stated inclusion criteria, yet the paper generalizes to ChatGPT-4o's overall chest X-ray capability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OpenAI ChatGPT interprets Radiological Images: GPT-4 as a Medical Doctor for a Fast Check-Up." pith.science (2026). https://pith.science/paper/2QTWNKJO

@misc{pith2026250106269,
  author       = {Pith},
  title        = {Pith review of: OpenAI ChatGPT interprets Radiological Images: GPT-4 as a Medical Doctor for a Fast Check-Up},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QTWNKJO}},
  note         = {Machine review of arXiv:2501.06269}
}

read the original abstract

OpenAI released version GPT-4 on March 14, 2023, following the success of ChatGPT, which was announced in November 2022. In addition to the existing GPT-3 features, GPT-4 can interpret images. To achieve this, the processing power and model have been significantly improved. The ability to process and interpret images goes far beyond the applications and effectiveness of artificial intelligence. In this study, we first explored the interpretation of radiological images in healthcare using artificial intelligence (AI). Then, we experimented with the image interpretation capability of the GPT-4. In this way, we addressed the question of whether artificial intelligence (AI) can replace a healthcare professional (e.g., a medical doctor) or whether it can be used as a decision-support tool that makes decisions easier and more reliable. Our results showed that ChatGPT is not sufficient and accurate to analyze chest X-ray images, but it can provide interpretations that can assist medical doctors or clinicians.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 9 canonical work pages

[1]

& Acharya, U

Alizadehsani, R., Khosravi, A., Roshanzamir, M., Abdar, M., Sarrafzadegan, N., Shafie, D., ... & Acharya, U. R. (2021). Coronary artery disease detection using artificial intelligence techniques: A survey of trends, geographical differences and diagnostic features 1991 –

work page 2021
[5]

https://github.com/ieee8023/covid-chestxray-dataset Harrer, S., Shah, P., Antony, B., & Hu, J

Github (2020). https://github.com/ieee8023/covid-chestxray-dataset Harrer, S., Shah, P., Antony, B., & Hu, J. (2019). Artificial intelligence for clinical trial design. Trends in pharmacological sciences, 40(8), 577-591. Huang, D., Yang, M., Wen, X., Xia, S., & Yuan, B. (2024). AI -driven drug dis covery:: accelerating the development of novel therapeutic...

work page 2020
[167]

Kiani, M., & Nasir, F. (2024). AI in Drug Discovery: Accelerating Pharmaceutical Research. International Journal of Advanced Engineering Technologies and Innovations, 1(1), 80-98. Koc, A. M., Bakiu, E., Gülas, G. E., & Koç, E. M. (2023). Sağlık İçin Yapay Zeka Teknolojilerinin Etik Tasarımı, Uygulanması Ve Kullanımı İçin Dikkat Edilecek Hususlar. Sağlık B...

work page 2024
[263]

Y., Balçık, P

Tarcan, G. Y., Balçık, P. Y., & Sebik, N. B. (2024). Türkiye ve Dünyada Sağlık Hizmetlerinde Yapay Zekâ. Mersin Üniversitesi Tıp Fakültesi Lokman Hekim Tıp Tarihi ve Folklorik Tıp Dergisi, 14(1), 50-60. Topol, E. (2023). Doctors, Get Ready for You r AI Assistants. WIRED. https://www.wired.co.uk/article/artificial-intelligence-medicine-machine-vision 15 Tr...

work page 2024
[439]

C., Monteferrante, E., Roy, M

Bélisle-Pipon, J. C., Monteferrante, E., Roy, M. C., & Couture, V. (2023). Artificial intelligence ethics has a black box problem. AI & SOCIETY, 1-16. Byskov, J., Maluka , S., Marchal, B., Shayo, E. H., Blystad, A., Bukachi, S., ... & Bloch, P. (2019). A systems perspective on the importance of global health strategy developments for accomplishing today’s...

arXiv 2023
[509]

& Sechopoulos, I

Rodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Broeders, M., Gennaro, G., Clauser, P., ... & Sechopoulos, I. (2019a). Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists. JNCI: Journal of the National Cancer Institute, 111(9), 916-922. Rodríguez-Ruiz, A., Krupinski, E., Mordang, J. J., Schilli...

work page 2019
[850]

Potla, R. T. (2021). Explainable AI (XAI) and its Role in Ethical Decision -Making. Journal of Science & Technology, 2(4), 151-174. Ray, P. P., & Majumder, P. (2023). The potential of ChatGPT to transform healthcare and address ethical challenges in artificial intelligence -driven medicine. Journal of Clinical Neurology (Seoul, Korea), 19(5),

work page 2021
[2014]

Lecler, A., Duron, L., & Soyer, P

İnsan Hakları Bağlamında Özel Hayatın Gizliliği Ve Korunması [Privacy and Protection of Private Life in the Context of Human Rights], Karamanoğlu Mehmetbey Üniversitesi Sosyal Ve Ekonomik Araştırmalar Dergisi, Cilt: 2014, Sayı: 3, ss 99-103, DOI: 10.18493/kmusekad.97442. Lecler, A., Duron, L., & Soyer, P. (2023). Revolutionizing radiology with GPT-based m...

work page doi:10.18493/kmusekad.97442 2023

Show all 10 references

[2019]

Advanced Technology Applications in Health [Sağlıkta İleri Teknoloji Uygulamaları] Book, Nobel Publishing House, p: 191-210 Karaarslan, E., & Konacaklı, E

Usage of Blockchain Technology in the Health Sector [Sağlık Sektöründe Blokzinciri Teknolojisinin Kullanımı]. Advanced Technology Applications in Health [Sağlıkta İleri Teknoloji Uygulamaları] Book, Nobel Publishing House, p: 191-210 Karaarslan, E., & Konacaklı, E. (2021). Dec...

2021
[2020]

AlJaberi, O

Computers in Biology and Medicine, 128, 104095. AlJaberi, O. A., Hussain, M., & Drake, P. R. (2020). A framework for measuring sustainability in healthcare systems. International journal of healthcare management. Amann, J., Blasimme, A., Vayena, E., Frey, D., Madai, V. I., & P...

2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

& Acharya, U

Alizadehsani, R., Khosravi, A., Roshanzamir, M., Abdar, M., Sarrafzadegan, N., Shafie, D., ... & Acharya, U. R. (2021). Coronary artery disease detection using artificial intelligence techniques: A survey of trends, geographical differences and diagnostic features 1991 –

work page 2021

[2] [5]

https://github.com/ieee8023/covid-chestxray-dataset Harrer, S., Shah, P., Antony, B., & Hu, J

Github (2020). https://github.com/ieee8023/covid-chestxray-dataset Harrer, S., Shah, P., Antony, B., & Hu, J. (2019). Artificial intelligence for clinical trial design. Trends in pharmacological sciences, 40(8), 577-591. Huang, D., Yang, M., Wen, X., Xia, S., & Yuan, B. (2024). AI -driven drug dis covery:: accelerating the development of novel therapeutic...

work page 2020

[3] [167]

Kiani, M., & Nasir, F. (2024). AI in Drug Discovery: Accelerating Pharmaceutical Research. International Journal of Advanced Engineering Technologies and Innovations, 1(1), 80-98. Koc, A. M., Bakiu, E., Gülas, G. E., & Koç, E. M. (2023). Sağlık İçin Yapay Zeka Teknolojilerinin Etik Tasarımı, Uygulanması Ve Kullanımı İçin Dikkat Edilecek Hususlar. Sağlık B...

work page 2024

[4] [263]

Y., Balçık, P

Tarcan, G. Y., Balçık, P. Y., & Sebik, N. B. (2024). Türkiye ve Dünyada Sağlık Hizmetlerinde Yapay Zekâ. Mersin Üniversitesi Tıp Fakültesi Lokman Hekim Tıp Tarihi ve Folklorik Tıp Dergisi, 14(1), 50-60. Topol, E. (2023). Doctors, Get Ready for You r AI Assistants. WIRED. https://www.wired.co.uk/article/artificial-intelligence-medicine-machine-vision 15 Tr...

work page 2024

[5] [439]

C., Monteferrante, E., Roy, M

Bélisle-Pipon, J. C., Monteferrante, E., Roy, M. C., & Couture, V. (2023). Artificial intelligence ethics has a black box problem. AI & SOCIETY, 1-16. Byskov, J., Maluka , S., Marchal, B., Shayo, E. H., Blystad, A., Bukachi, S., ... & Bloch, P. (2019). A systems perspective on the importance of global health strategy developments for accomplishing today’s...

arXiv 2023

[6] [509]

& Sechopoulos, I

Rodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Broeders, M., Gennaro, G., Clauser, P., ... & Sechopoulos, I. (2019a). Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists. JNCI: Journal of the National Cancer Institute, 111(9), 916-922. Rodríguez-Ruiz, A., Krupinski, E., Mordang, J. J., Schilli...

work page 2019

[7] [850]

Potla, R. T. (2021). Explainable AI (XAI) and its Role in Ethical Decision -Making. Journal of Science & Technology, 2(4), 151-174. Ray, P. P., & Majumder, P. (2023). The potential of ChatGPT to transform healthcare and address ethical challenges in artificial intelligence -driven medicine. Journal of Clinical Neurology (Seoul, Korea), 19(5),

work page 2021

[8] [2014]

Lecler, A., Duron, L., & Soyer, P

İnsan Hakları Bağlamında Özel Hayatın Gizliliği Ve Korunması [Privacy and Protection of Private Life in the Context of Human Rights], Karamanoğlu Mehmetbey Üniversitesi Sosyal Ve Ekonomik Araştırmalar Dergisi, Cilt: 2014, Sayı: 3, ss 99-103, DOI: 10.18493/kmusekad.97442. Lecler, A., Duron, L., & Soyer, P. (2023). Revolutionizing radiology with GPT-based m...

work page doi:10.18493/kmusekad.97442 2023