CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation

Bernhard Kainz; Bjoern Menze; Hadrien Reynaud; Ibrahim Ethem Hamamci; Sezgin Er; Suprosanna Shit

REVIEW 4 major objections 4 minor 2 cited by

CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CRG Score grades radiology reports by clinical findings, not word overlap, and can serve as a training reward for AI report generators.

desk verdict A clean, transparent metric whose claimed clinical alignment is asserted rather than shown — and whose ranking reduces to Youden's J. read the letter →

arxiv 2505.17167 v1 pith:64SUA3Z2 submitted 2025-05-22 cs.CL cs.CV

Ibrahim Ethem Hamamci , Sezgin Er , Suprosanna Shit , Hadrien Reynaud , Bernhard Kainz , Bjoern Menze This is my paper · ORCID

classification cs.CLcs.CV

keywords radiologyreportgenerationclinicalaccuracymetricclassimbalancedistribution-awareevaluationreinforcementlearningrewardCTLLM-basedfeatureextraction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CRG Score, a metric for evaluating radiology report generation that focuses only on clinically relevant abnormalities named in the reference report. It argues that standard natural-language metrics miss clinical correctness and that existing clinical accuracy metrics are distorted by class imbalance—high accuracy often just reflects many true negatives, and recall can reward verbose over-reporting. CRG Score balances penalties according to how rare abnormalities are in the dataset, so missing a finding in a low-prevalence setting costs more than a false positive, and vice versa. The authors show the score ranks four CT report generators differently from standard metrics and yields a fixed value (1/3) for trivial always-normal or always-abnormal outputs. They also propose the score as a reward function for reinforcement learning, so report generators can be trained to maximize clinical correctness rather than token-level likelihood.

What carries the argument

The central object is the CRG score itself, a distribution-aware clinical accuracy metric. Its construction has three steps: it restricts evaluation to clinically meaningful outcomes (ignoring true negatives), it equates the scores of the two extreme outputs—an empty report and an exhaustive report—to derive the weight ratio $(w_{TP} + w_{FN})/w_{FP} = (T-A)/A$, and it closes the remaining degree of freedom by assuming $w_{TP} = w_{FN}$. The resulting normalized score, CRG $= S_{\max}/(2S_{\max} - s)$, assigns 1/3 to trivial predictions. What makes it work is that the penalty for a false negative grows as the abnormality becomes rarer in the dataset, which matches the clinical intuition that missing an uncommon finding is more dangerous than an occasional false alarm.

What would settle it

Take a held-out set of CT reports with radiologist-assigned clinical-quality ratings, compute CRG for two systems—one that reports all major findings with many false positives, and one that omits a rare critical finding—and check whether CRG ranks them in the same order as the radiologists. If the metric prefers the system that misses the critical finding in a low-prevalence setting, the weight balance is wrong.

Watch

Extended reading notes

Core claim

The central claim is that CRG Score gives a fairer and more clinically truthful evaluation of generated radiology reports than existing metrics, and that the same score can serve as a training reward. The metric ignores true negatives, counts only abnormalities explicitly described in the reference report, and rebalances the weights of true positives, false negatives, and false positives by the label prevalence. It is defined so that an empty report and an exhaustive report receive the same raw score, which pins down the weight ratio; with the added assumption that the reward for a true positive equals the penalty for a false negative, the weights become $w_{TP} = w_{FN} = (T-A)/(2A)$ and $w_{FP} = 1$. The normalized score maps trivial solutions to 1/3 and higher values to better clinical performance. The paper demonstrates the metric on CT report generation with an 18-class abnormality labeler and reports that models with high lexical scores can be ranked differently once clinical balance is taken into account.

Load-bearing premise

The metric's balance between missing a finding and correctly reporting one is fixed by assuming the reward for a true positive equals the penalty for a false negative ($w_{TP} = w_{FN}$); if real clinical costs differ, the score's ranking of models could be miscalibrated.

Editorial extensions

If this is right

CRG can replace or complement NLG metrics for ranking radiology report generators, since it evaluates clinically relevant content rather than lexical overlap.
In low-prevalence settings, CRG penalizes missed abnormalities more heavily than false positives; in high-prevalence settings it does the reverse, so the same metric adapts to different data distributions.
CRG can serve as a reward function in reinforcement learning, allowing report generators to be optimized for clinical correctness, possibly combined with BLEU or cross-entropy for fluency.
CRG supports structured labels, so it can be extended beyond binary abnormality presence to include type, location, laterality, severity, or count.
The metric fixes the trivial-solution problem: always-normal and always-abnormal reports both receive CRG = 1/3, so a generator cannot inflate its score by predicting the majority class.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If CRG is adopted as an evaluation standard, model development will shift from mimicking reference style to optimizing detection of clinically relevant findings, which may change which architectures and training objectives win on leaderboards.
The metric's weight-balancing rule—making false negatives costlier when abnormalities are rare—is a sensible default but not the only clinically defensible one; a radiologist-grounded cost matrix could replace the $w_{TP} = w_{FN}$ assumption per use-case without changing the metric's structure.
The same distribution-aware balancing principle could transfer to other long-context generation tasks with skewed label distributions, such as pathology reports or discharge summaries, wherever true negatives are uninformative.
A direct test of clinical alignment would be comparing CRG rankings with radiologist preference judgments; if they diverge on a specific dataset, the weight ratio could be recalibrated rather than abandoning the metric.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this is a short MIDL paper proposing CRG Score, a class-imbalance-aware metric for radiology report generation. The formula is new and simple, and the implementation ships. But the headline claims — fairer, clinically aligned, reward-ready — are asserted, not demonstrated.

The good parts: CRG ignores true negatives, reweights penalties by label prevalence, and supports structured labels via a simple averaging scheme. The normalization trick that forces trivial reports to score 1/3 is neat and makes the metric easy to interpret. The code is public. As a definition, the math is correct and it is genuinely not a restatement of Green or standard accuracy.

The soft spots are real. The derivation in Section 2.1 equates empty and exhaustive reports, then assumes wTP = wFN with the justification that missing a finding and reporting one are equally costly. That is a value judgment, not a measured clinical cost. Worse, on a fixed test set this choice does not affect model ordering at all: after substitution CRG = 1/(3 - 2r + 2f), a monotone transform of r - f, i.e., Youden's J. So the actual ranking is equivalent to a decades-old imbalance-aware measure. The paper never compares against Youden's J, balanced accuracy, or Matthews correlation. The 'clinically aligned' claim rests on weighting all 18 CT-RATE abnormalities equally — a stable granuloma and an acute pneumothorax count the same — and on the unstated assumption that clinical utility is proportional to r - f. No radiologist preference data, no severity weighting, no RL experiment. Section 2.3 only flags the absence of structured-label evaluation, not the missing clinical validation. Minor: no error bars, though that is typical for a short venue.

This is not a takedown. The metric is clean, reimplementable, and honest about its scope. But as a claim of clinical alignment, the evidence is thin. The paper deserves peer review because the question it raises — what should a report-generation metric reward, and how should class imbalance be handled — matters, and the paper is a useful starting point. I would ask for a revision that adds comparisons to Youden's J and MCC, and either a small human-preference study or an explicit caveat that clinical alignment is a hypothesis, not a measured property.

Send it to review, with the expectation of heavy revision.

Referee Report

4 major / 4 minor

Summary. The paper proposes the CRG Score, a metric for radiology report generation that is intended to be distribution-aware and clinically aligned by weighting true positives, false negatives, and false positives according to label prevalence. The authors derive the weights from an axiom equating empty and exhaustive reports plus an assumption that the reward for a true positive equals the penalty for a false negative, then normalize the raw score so that trivial solutions receive 1/3. They evaluate the metric on CT-RATE with four models and report CRG scores alongside conventional clinical accuracy metrics.

Significance. If the claims were substantiated, a distribution-aware, clinically aligned metric for 3D radiology report generation would address a real need, since existing NLG and LLM-based metrics are poorly suited to clinical correctness. The paper provides a clear mathematical derivation, releases code, and evaluates on a relevant dataset, which are commendable. However, as shown in the major comments, the metric reduces to a monotone transformation of Youden's J, so the claimed distribution-aware weighting has no effect on model ranking. Additionally, the clinical-alignment claim is entirely unvalidated. The paper's practical significance therefore hinges on whether the normalization and framing add value beyond existing metrics, which is not demonstrated.

major comments (4)

[§2.1–2.2, Eqs. (1)–(3)] Substituting the derived weights wTP = wFN = (T−A)/(2A) and wFP = 1 into the raw score and the normalization in Eq. (3) yields CRG = 1/(3 − 2r + 2f), where r = TP/A and f = FP/(T−A). This is a monotone increasing function of Youden's J = r + specificity − 1. Consequently, the prevalence-dependent weights cancel out completely, and CRG induces exactly the same model ranking as Youden's J on any fixed test set. The claim that CRG 'balances penalties based on label distribution' is therefore not supported by the metric as defined. The authors should either modify the metric so that the weighting affects the ranking, or explicitly acknowledge and discuss the equivalence to Youden's J and reframe the contribution accordingly.
[Abstract and §2.3] The abstract and Section 2.3 state that CRG is 'clinically aligned' and 'serves as a clinically aligned reward function', but no clinical validation is provided. The paper reports no radiologist preference ratings, no severity weighting (e.g., a stable granuloma versus an acute pneumothorax), no comparison with clinician-preferred rankings, and no reinforcement-learning experiment showing that optimizing CRG improves clinically meaningful report content. The limitation paragraph in Section 2.3 only mentions the absence of structured-label evaluation; it does not address the missing clinical validation. The authors should either add clinical validation or substantially temper the claims.
[Table 2] The reported CRG scores have no error bars or statistical significance tests. The differences among CT2Rep, CT-CHAT, and Merlin (CRG 0.359, 0.368, and 0.352, respectively) may be within noise. The paper should include bootstrap confidence intervals or a significance test, and ideally evaluate on additional test sets to support the claim that CRG enables 'fairer, more robust evaluation'.
[Sections 2.2–2.3] The paper does not compare CRG against other imbalance-aware metrics such as Youden's J, balanced accuracy, Matthews correlation coefficient, or prevalence-weighted F-beta. Given the equivalence to Youden's J established in the first major comment, this comparison is essential to support the claim of 'fairer' evaluation. Without it, the reader cannot determine whether CRG offers any advantage over existing metrics beyond a different scaling.

minor comments (4)

[§2.2] The text says 'CRG = 1/3 = 0.3'; since 1/3 is approximately 0.333, the paper should write '0.333' or 'approximately 0.3' for precision.
[Table 2] The total number of labels T and the number of positive labels A are not reported in the text; adding these values would make the degree of class imbalance in the CT-RATE validation set explicit and help the reader interpret the metric.
[§2.3] The sentence 'CRG can serve as a reward function...' is speculative; the paper should clarify that this is a proposal not yet tested in an RL training scenario.
[§1] The abbreviation 'CA' is used for 'clinical accuracy' without being defined at first use in the Introduction; please define it.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CRG Score is a transparent metric definition; the clinical-alignment claim is an unvalidated assertion, not a circular derivation.

full rationale

The paper's derivation chain for CRG Score is self-contained in the sense that the score is defined, not inferred from a fitted target. Equation (1) fixes the FP weight by equating the empty and exhaustive report extremes, and Eq. (2) closes the remaining degree of freedom with an explicit assumption wTP = wFN. These are design choices made from stated clinical considerations; they do not use model outputs or a target ranking as inputs, so the resulting scores are not circular predictions. The use of the CT-RATE labeler (Hamamci et al., 2024a) is a self-citation, but it only supplies the demonstration label set; the metric is explicitly model-agnostic and 'can be paired with any recent LLM to extract report features,' so the self-citation is not load-bearing for the metric's definition. The paper's claim that CRG is 'clinically aligned' is asserted rather than demonstrated, since no radiologist-preference validation or reward-learning experiment is provided, but that is a validation gap rather than circularity. The assumption wTP = wFN is arbitrary, yet it enters as an explicit premise and does not smuggle the conclusion into the input. Overall, no step in the derivation reduces to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The metric rests on four explicit assumptions, two of which are introduced ad hoc to close the weight equations, and two which are domain assumptions about what should be scored. No free parameters are fitted to data; the weights derive from the test set's label prevalence.

assumptions (4)

ad hoc to paper An empty report and an exhaustive report should receive equal scores
This equality in Section 2.1 determines the weight ratio in Eq. (1). No clinical evidence is given to justify that a report listing every abnormality (and thus many false positives) is as bad as a report that says nothing.
ad hoc to paper The reward for a true positive equals the penalty for a false negative (wTP = wFN)
Stated in Section 2.1 as 'clinically grounded trade-off', this closes the underdetermined Eq. (1). In many clinical contexts, missing a critical finding is more harmful than correctly reporting one is beneficial, so this symmetry is not self-evident.
domain assumption True negatives carry no clinical value and can be ignored
The paper argues 3D reports omit normal findings and the CT-RATE labeler treats unmentioned findings as normal (Section 2.1). This assumption is plausible for chest CT but may fail when explicitly stating absence is clinically important, e.g., 'no acute intracranial hemorrhage'.
domain assumption Prevalence on the evaluation set is the correct basis for balancing penalties
The weights wTP and wFN depend on A/T from the test set (Eq. (2)). This assumes the test set's label distribution represents the clinical population; otherwise weights are miscalibrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation." pith.science (2026). https://pith.science/paper/64SUA3Z2

@misc{pith2026250517167,
  author       = {Pith},
  title        = {Pith review of: CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64SUA3Z2}},
  note         = {Machine review of arXiv:2505.17167}
}

read the original abstract

Evaluating long-context radiology report generation is challenging. NLG metrics fail to capture clinical correctness, while LLM-based metrics often lack generalizability. Clinical accuracy metrics are more relevant but are sensitive to class imbalance, frequently favoring trivial predictions. We propose the CRG Score, a distribution-aware and adaptable metric that evaluates only clinically relevant abnormalities explicitly described in reference reports. CRG supports both binary and structured labels (e.g., type, location) and can be paired with any LLM for feature extraction. By balancing penalties based on label distribution, it enables fairer, more robust evaluation and serves as a clinically aligned reward function.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression
cs.CV 2026-07 conditional novelty 6.0 of 10

ORCA compresses 3D CT tokens into organ-guided connected regions with sinusoidal centroid encoding, outperforming grid average and other compressors at matched budgets.
Structured Spectral Graph Representation Learning for Multi-label Abnormality Analysis from 3D CT Scans
cs.CV 2025-10 conditional novelty 4.0 of 10

A graph-of-slice-triplets encoder with spectral convolution outperforms 3D CNN/Transformer baselines on multi-label chest CT abnormality classification and transfers to report generation and abdominal CT.

Reference graph

Works this paper leans on

9 extracted references · 6 canonical work pages · cited by 2 Pith papers

[1]

Merlin: A vision language foundation model for 3d computed tomography

Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs--3, 2024

work page 2024
[2]

Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes

Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical image analysis, 67: 0 101857, 2021

work page 2021
[3]

Developing generalist foundation models from a multimodal dataset for 3d computed tomography

Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Omer Faruk Durugol, Bastian Wittmann, Tamaz Amiranashvili, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834, 2024 a

arXiv 2024
[4]

Ct2rep: Automated radiology report generation for 3d medical imaging

Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. Ct2rep: Automated radiology report generation for 3d medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 476--486. Springer, 2024 b

work page 2024
[5]

Generatect: Text-conditional generation of 3d chest ct volumes

Ibrahim Ethem Hamamci, Sezgin Er, Anjany Sekuboyina, Enis Simsar, Alperen Tezcan, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Furkan Almas, Irem Do g an, Muhammed Furkan Dasdelen, et al. Generatect: Text-conditional generation of 3d chest ct volumes. In European Conference on Computer Vision, pages 126--143. Springer, 2024 c

work page 2024
[6]

Unetr: Transformers for 3d medical image segmentation

Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574--584, 2022

work page 2022
[7]

Llava-med: Training a large language-and-vision assistant for biomedicine in one day

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 0 28541--28564, 2023

work page 2023
[8]

Green: Generative radiology report evaluation and error notation

Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. Green: Generative radiology report evaluation and error notation. arXiv preprint arXiv:2405.03595, 2024

arXiv 2024

Show all 9 references

[9]

Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308.02463, 2023

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Merlin: A vision language foundation model for 3d computed tomography

Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs--3, 2024

work page 2024

[2] [2]

Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes

Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical image analysis, 67: 0 101857, 2021

work page 2021

[3] [3]

Developing generalist foundation models from a multimodal dataset for 3d computed tomography

Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Omer Faruk Durugol, Bastian Wittmann, Tamaz Amiranashvili, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834, 2024 a

arXiv 2024

[4] [4]

Ct2rep: Automated radiology report generation for 3d medical imaging

Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. Ct2rep: Automated radiology report generation for 3d medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 476--486. Springer, 2024 b

work page 2024

[5] [5]

Generatect: Text-conditional generation of 3d chest ct volumes

Ibrahim Ethem Hamamci, Sezgin Er, Anjany Sekuboyina, Enis Simsar, Alperen Tezcan, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Furkan Almas, Irem Do g an, Muhammed Furkan Dasdelen, et al. Generatect: Text-conditional generation of 3d chest ct volumes. In European Conference on Computer Vision, pages 126--143. Springer, 2024 c

work page 2024

[6] [6]

Unetr: Transformers for 3d medical image segmentation

Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574--584, 2022

work page 2022

[7] [7]

Llava-med: Training a large language-and-vision assistant for biomedicine in one day

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 0 28541--28564, 2023

work page 2023

[8] [8]

Green: Generative radiology report evaluation and error notation

Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. Green: Generative radiology report evaluation and error notation. arXiv preprint arXiv:2405.03595, 2024

arXiv 2024