Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings

Diego Machado Reyes; Ge Wang; Joy T. Wu; Mannudeep K. Kalra; Parisa Kaviani; Pingkun Yan; Razi Mahmood; Tanveer Syeda-Mahmood

REVIEW 5 major objections 5 minor 23 references

Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new metric that pairs fine-grained textual finding overlap with anatomical grounding in the image detects wrong findings, wrong locations, and wrong severity in AI chest X-ray reports, which text-only metrics like BLEU barely register.

desk verdict Good composite-metric idea, but the pattern-completion step makes the location-sensitivity claim unreliable, and the paper needs cleanup before it can be trusted. read the letter →

arxiv 2412.01031 v3 pith:NQNRRVKS submitted 2024-12-02 cs.CL cs.AIcs.CV

Razi Mahmood , Pingkun Yan , Diego Machado Reyes , Ge Wang , Mannudeep K. Kalra , Parisa Kaviani , Joy T. Wu , Tanveer Syeda-Mahmood This is my paper · ORCID

classification cs.CLcs.AIcs.CV

keywords chestX-rayreportevaluationfactualerrordetectionfine-grainedfindingpatternsphrasalgroundinggenerativeAIradiologyreportsqualitymetricintersectionoverunionclinicalNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative AI can write fluent chest X-ray reports that look correct but contain wrong findings, missed findings, or misplaced anatomy. This paper proposes the RQ score, an automatic evaluator that extracts fine-grained finding patterns (type, negation, anatomy, laterality, severity) from a ground-truth report and a generated report, localizes each finding's anatomical region in the X-ray, and combines the textual F1 with the spatial overlap (mean IOU). The paper's central claim is that this combined score is sensitive to the factual errors that matter clinically, whereas text-only metrics such as BLEU remain nearly flat when location or severity is wrong. If the claim holds, hospitals and model developers can fact-check AI radiology reports automatically, per finding and per location, without waiting for a radiologist.

What carries the argument

The load-bearing object is the fine-grained finding pattern (FFL), written $F_i = T_i|N_i|C_i|A_i|L_i|S_i$, where $T_i$ is the finding type, $N_i$ is present or absent, $C_i$ is the normalized core finding, and $A_i$, $L_i$, $S_i$ are anatomy, laterality, and severity. Prefixes of these tuples are matched between ground truth and generated reports to compute the FFL F1-score, which grades the textual description at increasing granularity. The anatomy slot is grounded in the image by predicted bounding boxes for anatomical regions, and a bipartite graph with intersection-over-union edge weights is solved by maximum matching to produce the mean IOU. The report quality score is formed as $RQ = F1 + MIOU$, which is what makes a single miss—say, the right finding in the wrong lobe—count against the report twice: once textually, once spatially.

What would settle it

Recompute the 500-report sensitivity study with clinician-corrected FFL patterns for every generated report; if the corrected RQ sensitivities fall to the 0.1–0.3 range of BLEU, the reported advantage is an artifact of the parser. Independently, compute RQ on a set of real AI reports that radiologists have flagged as containing or not containing a factual error; the claim stands only if RQ separates the flagged from unflagged reports at a rate similar to the sensitivity table.

Watch

Extended reading notes

Core claim

On a gold-standard set of 439 chest X-rays with clinician-validated reports and anatomical bounding boxes, the authors show that the RQ score—defined in their Eq. 8 as $F1 + MIOU$—reacts to deliberately introduced errors with sensitivity 0.5 for finding errors, 0.4 for location errors, and 0.39 for severity errors, while BLEU gives 0.3, 0.1, and 0.1 on the same 500 perturbed reports. They also find that text-only metrics mis-calibrate quality: BLEU and BERTScore underestimate how much of the ground-truth content is present, and RadGraph F1 overestimates it because its clinical entity model covers only 14 findings. The RQ scores for three tested report generators fall in a middle range (combined scores roughly 0.35–0.46), reflecting both the textual match and the spatial correctness of the findings.

Load-bearing premise

The load-bearing premise is that the lexicon-based parser extracts the right finding type, negation, anatomy, laterality, and severity from both ground-truth and AI-generated reports; the paper cites a prior error rate of about 3% and does not re-validate the parser on these 439 reports or these generators, so if the parser mis-parses, both the FFL F1 and the mean IOU are computed on the wrong findings.

Editorial extensions

If this is right

Adopting RQ as an evaluation metric would let model developers rank generators by factual fidelity, not surface similarity; in the paper's head-to-head comparison the three generators separate more clearly under RQ than under BLEU.
A low RQ can be traced back to specific FFL mismatches and low-overlap anatomical boxes, so the metric doubles as a per-finding diagnostic instead of a single opaque number.
Because location and severity errors move RQ by 0.4 and 0.39 respectively, versus 0.1 for BLEU, automated screening of generated reports can catch these clinically significant errors without a radiologist in the loop.
The score can be computed automatically on large report collections, making it practical for continuous monitoring of clinical AI systems rather than one-off validation studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The sensitivity test is built from synthetic perturbations of ground-truth reports; a stronger test the authors do not report would score real AI-generated reports against radiologist-identified factual errors and check whether the same RQ advantage appears.
The metric's structure suggests a natural extension: return a per-finding error vector (missing finding, wrong side, wrong severity) rather than one score, which would convert the evaluator into a debugging tool for report generators.
Transfer to other imaging modalities is not immediate, because the anatomy grounding depends on a fixed set of standardized chest X-ray regions; a CT or MRI version would need a comparable anatomical atlas.
The published parser accuracy of about 3% error was measured on earlier reports; re-measuring it on modern generative outputs would be the cheapest way to stress-test the sensitivity results.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is sensible: combine fine-grained textual finding patterns (FFL F1) with anatomical grounding (MIOU) to get a report quality score that is more sensitive to location and severity errors than BLEU or BERT. That combination, evaluated on a public dataset with no fitted parameters, is a legitimate new contribution, and the authors deserve credit for comparing against a reasonable set of baselines and for including a synthetic perturbation study.

But the load-bearing flaw is the FFL pattern completion. As the paper itself states, the extraction step inserts anatomical locations from domain knowledge even when the sentence does not mention them (e.g., "alveoli" for an alveolar finding). Since the same completion is applied to both ground truth and generated reports, a generated report that omits a location can be parsed to the same completed FFL as one that states it correctly. The metric then gives full credit for the location because the completed pattern includes it. The synthetic sensitivity test does not fix this: it perturbs explicit location tokens in otherwise complete reports, so it only shows sensitivity to token changes, not to real-world omissions. This directly undercuts the paper's main claim that RQ is more sensitive to fine-grained factual errors.

There are also smaller issues. Table 2's FFL F1 values do not match Table 1 (0.440 vs. 0.331/0.486/0.511 for RGRG). Equation 8 omits the /2 that the table uses to compute the combined score. No error bars are given in Table 3. And there is no human validation: the metric is not compared against clinician judgments, which would be the natural check for an evaluation metric.

The parser accuracy is cited from prior work rather than re-measured on these specific GPT-4, XrayGPT, and RGRG outputs. That would be important even without the completion problem, because parser errors propagate into both FFL F1 and MIOU.

Still, the paper is not a waste of time. The composite score idea is worth exploring, and the problems are identifiable and potentially fixable. If the authors could either disable or clearly separate pattern completion, or show that the metric still detects omissions despite it, and if they fixed the small inconsistencies and added human validation, the paper could be solid. As it stands, I would not trust the sensitivity numbers.

For a serious referee: yes, I would send it. The flaws are substantive but not beyond repair, and the topic matters for the medical imaging NLP community. For my own work, I would not cite the sensitivity results until the completion issue is resolved.

Referee Report

5 major / 5 minor

Summary. The paper proposes a new automatic evaluation metric, RQ, for chest X-ray radiology reports. RQ combines a fine-grained finding-level F1 score (FFL F1) with a mean intersection-over-union (MIOU) score obtained by grounding findings to anatomical bounding boxes on the image. The metric is evaluated on 439 ChestImagenome reports compared against reports generated by RGRG, XrayGPT, and GPT-4, and is tested for sensitivity using 500 synthetic perturbed reports. The central claim is that RQ is more sensitive to factual errors—particularly location and severity errors—than existing text-only metrics such as BLEU, BERTScore, and RadGraph.

Significance. If the proposed metric is validated, it would be a useful contribution because existing text-only metrics do not explicitly capture anatomical location, laterality, and severity, and RQ additionally incorporates visual grounding. The paper compares RQ against independent baselines on a public dataset and the metric has no fitted parameters, which are strengths. The idea of combining a structured clinical finding representation with phrasal grounding is sensible and potentially valuable for automatic fact-checking of radiology reports. However, the current evaluation has several gaps—inconsistent numeric definitions, an unaddressed pattern-completion issue, and the lack of human validation—so the central sensitivity claim is plausible but not rigorously established.

major comments (5)

[Section 3, Eq. (8)] The definition RQ(G,P) = F1(G,P) + MIOU(G,P) in Eq. (8) contradicts the preceding sentence, which says the final quality score is the average of the F1 and mean IOU scores, and it also contradicts Table 2, where the Combined column for RGRG is 0.463 = (0.440+0.487)/2. As written, Eq. (8) gives values exactly twice the average, and it is unclear whether the sensitivity values in Table 3 were computed with or without the missing factor of 1/2. Please correct the equation and state the exact formula used for all reported numbers.
[Tables 1 and 2] The FFL F1-score for the same methods is reported inconsistently. Table 1 gives 'All' FFL F1 of 0.331, 0.223, and 0.242 for RGRG, XrayGPT, and GPT-4, while Table 2 gives 0.440, 0.391, and 0.326 for the 'Avg. FFL F1-score' of the same three systems. No explanation is given for which prefix granularity is used in Table 2; if the comparison with BLEU, BERT, and RadGraph uses a different FFL definition than Table 1, the reader cannot infer the source of the discrepancy. Please reconcile the two tables or label the metric definitions in each.
[Section 2, FFL extraction, and Table 3] The pattern completion step explicitly inserts anatomical locations from domain knowledge 'even when not specified in the report sentence' (e.g., alveoli for an alveolar finding). Because Eqs. (2)–(7) compute FFL F1 and MIOU on these completed patterns, a generated report that omits the location of an opacity can be completed to the same FFL pattern as one that correctly states 'opacity in the right upper lobe.' The sensitivity study perturbs explicit location and severity tokens in otherwise complete reports, so it does not test the metric's ability to detect omissions. The cited ~3% parser error is from prior work and is not re-measured on GPT-4, XrayGPT, or RGRG outputs; even a low token-level error rate would not reveal systematic insertion behavior. To support the claim that RQ captures 'absence of findings' and location errors, either match only modifiers actually stated in the generated text, or measure the completion rate on the three generators' outputs and demonstrate that RQ drops when locations are omitted.
[Section 4, Table 3] The sensitivity comparison is reported as single numbers without error bars, significance tests, or the number of perturbations per error category. The 500 synthetic reports are generated by token-level edits of ground truth reports and may not reflect the error distribution of real generators; without confidence intervals and a more realistic error model, the claim that RQ is more sensitive than BLEU, BERT, and RadGraph is not rigorously established.
[Section 4 and Conclusions] No human validation is reported. The paper does not compare RQ with clinician ratings of report quality or with human judgments of factual errors, so the statement in the Conclusions that RQ is 'suitable as an evaluation metric for fact-checking of radiology reports' is not directly supported. At minimum, a correlation study with expert quality ratings on a sample of generated reports would substantiate the metric's validity.

minor comments (5)

[Section 3, Eq. (6)] The notation in Eq. (6) is unclear: Iki sums over index i, but the right-hand side defines an intersection-over-union between two bounding boxes, and the meaning of the sum is not explained. Please clarify the indexing for the bipartite matching.
[Section 3, text after Eq. (4)] There is a typo, 'computer F1 score', which should be 'compute F1 score'; also, Eq. (3) has mismatched parentheses in 'Wj(FGi' and 'Wj(FP k'.
[Section 4, Table 3] The table does not indicate how many perturbations were generated for each error type (finding, location, severity) or how the edits were sampled. Reporting the perturbation recipe would improve reproducibility.
[References] Reference [6] is incomplete: it lists only 'Kishore Papineni et al.' and a URL, without the full author list and publication venue. Please provide complete citation information.
[Section 5, Discussion] The phrase 'exploited the notation of standardized locations' should likely be 'exploited the notion of standardized locations'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RQ is a deterministic, non-fitted combination of independently published component extractors, and the sensitivity comparison is benchmarked against external baselines on public data.

full rationale

The paper's derivation chain does not reduce to its inputs. Equation 8 defines RQ as the average of an FFL-prefix F1 (Eqs. 2-5) and a mean IoU from bipartite matching of anatomical boxes (Eqs. 6-7); neither component is fitted to the reports being scored, and neither is defined in terms of RQ itself. The FFL extractor and bounding-box detector are borrowed from prior publications by the same group ([2,16,13,17]), but those are separate component systems with reported external accuracy (about 3% extraction error; 0.896 precision / 0.881 recall for the detector), and the paper evaluates RQ against independent baselines (BLEU, BERTScore, RadGraph) on the public ChestImagenome/MIMIC gold standard. The sensitivity experiment (Section 4) perturbs the actual report text and re-extracts FFL patterns and boxes, so the reported score changes are measured outcomes rather than fitted values. The pattern-completion step that inserts canonical anatomical locations is a genuine construct-validity threat for detecting omitted locations, but it is a limitation of the component representation, not a circular reduction of RQ to its own output. No self-referential or definitional equivalence was found in the claimed derivation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three pre-existing components treated as trusted inputs: the authors' FFL parser, the authors' anatomy detector, and the ChestImagenome ground truth. The equal weighting of text and image components is chosen by hand. No new entities are invented.

free parameters (1)

Equal weighting alpha for textual and visual components = 0.5
Eq. 8 adds FFL F1 and MIOU with equal weight and Table 2 averages them; no justification or ablation for this weighting is given, and a different weight would change the combined score and possibly the ranking.

assumptions (4)

domain assumption The FFL parser [2,12,16] accurately and completely extracts fine-grained finding patterns (type, negation, anatomy, laterality, severity) from free-text reports.
Section 2 'Extraction of FFL patterns' builds the metric on this parser and cites ~3% error from prior work; no re-evaluation on the ChestImagenome subset or on generated reports is provided.
domain assumption The Faster RCNN detector [13,17] accurately localizes the 36 anatomical regions in the evaluation images.
Section 2 'Extraction of anatomical locations from images' relies on prior precision 0.896/recall 0.881; errors in box placement directly affect the MIOU component of RQ.
domain assumption ChestImagenome ground truth reports and their FFL annotations are correct and comprehensive for the 439 images.
Section 4 'Dataset' treats these as gold standard; if ground truth FFL patterns miss or mislabel findings, the computed F1 and IOU are biased.
ad hoc to paper The 500 synthetic perturbed reports in Table 3 are representative of real factual errors made by report generators.
The perturbation procedure is described only briefly as negation reversal, substitutions, and location/severity alterations; no details on counts or distributions, so the sensitivity numbers may not transfer to real errors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings." pith.science (2026). https://pith.science/paper/NQNRRVKS

@misc{pith2026241201031,
  author       = {Pith},
  title        = {Pith review of: Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQNRRVKS}},
  note         = {Machine review of arXiv:2412.01031}
}

read the original abstract

Several evaluation metrics have been developed recently to automatically assess the quality of generative AI reports for chest radiographs based only on textual information using lexical, semantic, or clinical named entity recognition methods. In this paper, we develop a new method of report quality evaluation by first extracting fine-grained finding patterns capturing the location, laterality, and severity of a large number of clinical findings. We then performed phrasal grounding to localize their associated anatomical regions on chest radiograph images. The textual and visual measures are then combined to rate the quality of the generated reports. We present results that compare this evaluation metric with other textual metrics on a gold standard dataset derived from the MIMIC collection and show its robustness and sensitivity to factual errors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

[1]

Figure 1e shows a sample re- port using GPT-4 [5] on the chest X-ray image shown on Fig- ure 1a

INTRODUCTION With the evolution of AI models, it is now possible to produce realistic-looking natural language radiology reports, particu- larly for chest X-rays [1, 2, 3, 4]. Figure 1e shows a sample re- port using GPT-4 [5] on the chest X-ray image shown on Fig- ure 1a. While this appears good on surface, upon closer ex- amination and comparing to the g...

work page
[2]

Given a chest X-ray image and its associated ground truth report, we first extract fine-grained finding (FFL) patterns from the ground truth report as described in[12]

OVERALL APPROACH Our overall approach to evaluating report quality is illustrated in Figure 2. Given a chest X-ray image and its associated ground truth report, we first extract fine-grained finding (FFL) patterns from the ground truth report as described in[12]. This creates a structured description of the report us- ing a normalized vocabulary for findi...

arXiv 2025
[3]

DEVELOPING REPORT EV ALUA TION SCORE We now describe our clinical accuracy score using the struc- tured representation of the findings in terms of FFL patterns and their phrasal grounding. Specifically, given a ground truth radiology report G and a predicted automated report P, we ex- tract FFL pattern set from sentences within these reports as FG and FP ...

work page
[4]

RESULTS We now present results of applying the quality score to assess report quality on a benchmark dataset of chest X-ray images with validated ground truth reports. Dataset: For our experiments, we selected the gold dataset of 439 chest x-rays and their ground truth reports from the pub- licly available clinician validated ChestImagenome[13] col- lecti...

work page
[5]

CONCLUSIONS In this paper, we present a new approach to evaluating the quality of generated chest X-ray radiology reports. Our approach captured fine-grained finding patterns along with phrasal grounding of findings and is shown to be sensitive to factual errors in radiology reports making it suitable as an evaluation metric for fact-checking of radiology reports

work page
[6]

A survey on au- tomatic generation of medical imaging reports based on deep learning,

Ting Pang, Peigao Li, and Lijie Zhao, “A survey on au- tomatic generation of medical imaging reports based on deep learning,” BioMedical Engineering OnLine , vol. 22, pp. 48, 2023

work page 2023
[7]

Chest x-ray report generation through fine-grained label learn- ing,

Tanveer Syeda-Mahmood, Ken C L Wong, Yaniv Gur, Joy T Wu, Ashutosh Jadhav, Satyananda Kashyap, Alexandros Karargyris, Anup Pillai, Arjun Sharma, Ali Bin Syed, Orest Boyko, and Mehdi Moradi, “Chest x-ray report generation through fine-grained label learn- ing,” in MICCAI-2020, 2020

work page 2020
[8]

Retrieval-based chest x-ray report generation using a pre-trained con- trastive language-image model,

Mark Endo, Rayan Krishnan, Viswesh Krishna, An- drew Y . Ng, and Pranav Rajpurkar, “Retrieval-based chest x-ray report generation using a pre-trained con- trastive language-image model,” Proceedings of Ma- chine Learning Research , vol. 158, pp. 209–219, 11 2021

work page 2021

Show all 23 references

[9]

Automated generation of accurate & fluent medical x- ray reports,

Hoang T.N. Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong, and Li Cheng, “Automated generation of accurate & fluent medical x- ray reports,” EMNLP 2021 - 2021 Conference on Em- pirical Methods in Natural Language Processing, Pro- ceedings, pp. 355...

2021
[10]

Evaluation of gpt-4 for chest x-ray impression generation: A reader study on performance and perception,

Sebastian Ziegelmayer, Alexander W. Marka, Nicolas Lenhart, Nadja Nehls, Stefan Reischl, Felix Harder, Andreas Sauter, Marcus Makowski, Markus Graf, and Joshua Gawlitza, “Evaluation of gpt-4 for chest x-ray impression generation: A reader study on performance and perception,” ...

2023
[11]

Bleu: a method for automatic evaluation of machine translation,

Kishore Papineni et al., “Bleu: a method for automatic evaluation of machine translation,” https://www.aclweb.org/anthology/P02-1040.pdf, 2002

2002
[12]

ROUGE: A package for automatic eval- uation of summaries,

Chin-Yew Lin, “ROUGE: A package for automatic eval- uation of summaries,” in Text Summarization Branches Out, Barcelona, Spain, July 2004, pp. 74–81, Associa- tion for Computational Linguistics

2004
[13]

Bertscore: Evaluating text generation with BERT,

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi, “Bertscore: Evaluating text generation with BERT,” CoRR, vol. abs/1904.09675, 2019

1904 arXiv
[14]

Radgraph: Extracting clinical entities and relations from radiology reports,

Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q. H. Truong, Du Nguyen Duong, Tan Bui, Pierre J. Chambon, Yuhao Zhang, Matthew P. Lungren, An- drew Y . Ng, Curtis P. Langlotz, and Pranav Rajpurkar, “Radgraph: Extracting clinical entities and relations from radiology repor...

2021 arXiv
[15]

G-eval: Nlg evalua- tion using gpt-4 with better human alignment,

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, “G-eval: Nlg evalua- tion using gpt-4 with better human alignment,” EMNLP 2023 - 2023 Conference on Empirical Methods in Natu- ral Language Processing, Proceedings, pp. 2511–2522, 2023

2023
[16]

Evaluating progress in automatic chest x-ray radiology report generation,

Feiyang Yu, Mark Endo, Rayan Krishnan, Curtis P Lan- glotz, Vasantha Kumar Venugopal, and Rajpurkar Cor- respondence, “Evaluating progress in automatic chest x-ray radiology report generation,” Patterns, vol. 4, pp. 100802, 2023

2023
[17]

Ai accelerated human-in-the-loop struc- turing of radiology reports,

J. Wu et al., “Ai accelerated human-in-the-loop struc- turing of radiology reports,” in Proc. American Medical Association Annual Symposium (AMIA) , Nov. 2020, p. 1305–1314

2020
[18]

Chest imagenome dataset for clinical reasoning,

Joy T. Wu, Nkechinyere N. Agu, Ismini Lourentzou, Ar- jun Sharma, Joseph A. Paguio, Jasper S. Yao, Edward C. Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, Leo A. Celi, and Mehdi Moradi, “Chest imagenome dataset for clinical reasoning,” 7 2021

2021
[19]

Xraygpt: Chest radiographs summarization using medical vision-language models,

Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad An- wer, Salman Khan, Jorma Laaksonen, and Fahad Shah- baz Khan, “Xraygpt: Chest radiographs summarization using medical vision-language models,” 6 2023

2023
[20]

Interactive and explainable region-guided ra- diology report generation,

Tim Tanida, Philip M¨uller, Georgios Kaissis, and Daniel Rueckert, “Interactive and explainable region-guided ra- diology report generation,” in CVPR, 2023

2023
[21]

Extracting and learning fine- grained labels from chest radiographs,

T. Syeda-Mahmood et al., “Extracting and learning fine- grained labels from chest radiographs,” in Proc. Amer- ican Medical Association Annual Symposium (AMIA) , Nov. 2020, p. 1190–1199

2020
[22]

Automatic bounding box annotation of chest x-ray data for localization of abnormalities,

Joy Wu, Yaniv Gur, Alexandros Karargyris, Ali Bin Syed, Orest Boyko, Mehdi Moradi, and Tanveer Syeda- Mahmood, “Automatic bounding box annotation of chest x-ray data for localization of abnormalities,” Pro- ceedings - International Symposium on Biomedical Imaging, vol. 2020-Ap...

2020
[23]

Mimic-cxr: A large publicly available database of labeled chest radiographs,

A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. y. Deng, R. G. Mark, and S. Horng, “Mimic-cxr: A large publicly available database of labeled chest radiographs,” arXiv preprint arXiv:1901.07042, 2019

1901 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Figure 1e shows a sample re- port using GPT-4 [5] on the chest X-ray image shown on Fig- ure 1a

INTRODUCTION With the evolution of AI models, it is now possible to produce realistic-looking natural language radiology reports, particu- larly for chest X-rays [1, 2, 3, 4]. Figure 1e shows a sample re- port using GPT-4 [5] on the chest X-ray image shown on Fig- ure 1a. While this appears good on surface, upon closer ex- amination and comparing to the g...

work page

[2] [2]

Given a chest X-ray image and its associated ground truth report, we first extract fine-grained finding (FFL) patterns from the ground truth report as described in[12]

OVERALL APPROACH Our overall approach to evaluating report quality is illustrated in Figure 2. Given a chest X-ray image and its associated ground truth report, we first extract fine-grained finding (FFL) patterns from the ground truth report as described in[12]. This creates a structured description of the report us- ing a normalized vocabulary for findi...

arXiv 2025

[3] [3]

DEVELOPING REPORT EV ALUA TION SCORE We now describe our clinical accuracy score using the struc- tured representation of the findings in terms of FFL patterns and their phrasal grounding. Specifically, given a ground truth radiology report G and a predicted automated report P, we ex- tract FFL pattern set from sentences within these reports as FG and FP ...

work page

[4] [4]

RESULTS We now present results of applying the quality score to assess report quality on a benchmark dataset of chest X-ray images with validated ground truth reports. Dataset: For our experiments, we selected the gold dataset of 439 chest x-rays and their ground truth reports from the pub- licly available clinician validated ChestImagenome[13] col- lecti...

work page

[5] [5]

CONCLUSIONS In this paper, we present a new approach to evaluating the quality of generated chest X-ray radiology reports. Our approach captured fine-grained finding patterns along with phrasal grounding of findings and is shown to be sensitive to factual errors in radiology reports making it suitable as an evaluation metric for fact-checking of radiology reports

work page

[6] [6]

A survey on au- tomatic generation of medical imaging reports based on deep learning,

Ting Pang, Peigao Li, and Lijie Zhao, “A survey on au- tomatic generation of medical imaging reports based on deep learning,” BioMedical Engineering OnLine , vol. 22, pp. 48, 2023

work page 2023

[7] [7]

Chest x-ray report generation through fine-grained label learn- ing,

Tanveer Syeda-Mahmood, Ken C L Wong, Yaniv Gur, Joy T Wu, Ashutosh Jadhav, Satyananda Kashyap, Alexandros Karargyris, Anup Pillai, Arjun Sharma, Ali Bin Syed, Orest Boyko, and Mehdi Moradi, “Chest x-ray report generation through fine-grained label learn- ing,” in MICCAI-2020, 2020

work page 2020

[8] [8]

Retrieval-based chest x-ray report generation using a pre-trained con- trastive language-image model,

Mark Endo, Rayan Krishnan, Viswesh Krishna, An- drew Y . Ng, and Pranav Rajpurkar, “Retrieval-based chest x-ray report generation using a pre-trained con- trastive language-image model,” Proceedings of Ma- chine Learning Research , vol. 158, pp. 209–219, 11 2021

work page 2021