Pith. sign in

REVIEW 6 major objections 6 minor 50 references

Semantically Informed Salient Regions Guided Radiology Report Generation

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A cross-modal saliency map that marks clinically critical patches in chest X-rays, guiding both masked image modeling and report decoding, lifts radiology report generation on IU-Xray and MIMIC-CXR.

desk verdict A solid RRG paper with a real methodological twist and consistent gains, but the central 'semantically informed' claim is never isolated from the saliency mechanism itself. read the letter →

arxiv 2507.11015 v1 pith:E4GR4XW3 submitted 2025-07-15 cs.CV cs.AI

Zeyi Hou , Zeqiang Wei , Ruixin Yan , Ning Lang , Xiuzhuang Zhou This is my paper · ORCID
classification cs.CVcs.AI
keywords radiologyreportgenerationchestX-raysalientregiondetectioncross-modalcontrastivelearningmaskedimagemodelingclinicalefficacymetricsIU-XrayMIMIC-CXR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Radiology reports are dominated by normal descriptions, while the abnormalities that determine clinical value are small and sparse, so models that produce fluent text often miss the findings that matter. The paper proposes SISRNet, which first learns a saliency map over image patches from fine-grained alignment between chest X-ray images and their paired reports, then uses that map twice: to increase the masking probability of salient patches during masked image modeling, and to inject a saliency-weighted visual token into the report decoder. The central claim is that this explicit prioritization of semantically informed regions counters the normal-heavy data bias and yields reports that are both fluent and measurably more clinically accurate than prior specialized models. On IU-Xray and MIMIC-CXR, SISRNet reports gains on n-gram metrics and on micro-averaged clinical efficacy metrics, with the largest example being MIMIC-CXR CE F1 of 0.509 versus 0.391 for the best prior method listed.

What carries the argument

The load-bearing object is the semantically informed saliency map $M_S \in \mathbb{R}^{1\times N_v}$, defined as the cosine similarity between each local vision patch embedding and the max-pooled global image embedding in a common semantic space learned with fine-grained cross-modal alignment (FCSA) and the mapping-before-aggregation (MbA) strategy. Top k% patches form salient regions $S$ that carry lesion-like semantics. The map does two jobs: it raises the masking probability of those patches in masked image modeling via $p_i \leftarrow p_i + \mathbb{I}(x_i^v \in S)\varphi$, and it produces a discriminative decoder token $w' = \mathrm{Norm}(M_S E_I)$ that injects saliency information into the language model's attention.

What would settle it

The central claim would be falsified by a controlled study on MIMIC-CXR where the same backbone replaces SISRNet's saliency map with random patch scores, CAM-based maps, and anatomical atlas priors under identical splits and seeds: if the saliency map does not beat those alternatives on micro CE F1 by more than seed noise, the saliency-guided component is not the cause of the gains. A sharper test is to permute the top-k patch identities and check whether CE F1 hardly moves.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that cross-modal semantics can localize the clinically decisive parts of a chest X-ray without any region annotation, and that routing those regions through both the image encoder and the language decoder is what improves report quality. Salient regions are defined as the top k% of patches whose local embeddings are most similar to the max-pooled global image embedding inside a common image-text space learned with a bidirectional contrastive loss, and the resulting saliency map biases masking in a masked autoencoder and is multiplied with image patch features to form an extra decoder token. Ablations on MIMIC-CXR attribute the gain to both pathways: saliency-guided masking raises micro CE F1 from 0.294 to 0.499, saliency-guided language modeling to 0.380, and the combined system to 0.509. The paper also argues that task-specific SISRNet outperforms several medical large language models on MIMIC-CXR under the comparison protocol used.

Load-bearing premise

The load-bearing premise is that the saliency map, computed as the cosine similarity between each local patch feature and the max-pooled global image feature in a contrastively learned space, reliably marks the medically critical regions; if normal anatomy dominates that global feature or the contrastive objective spreads semantics across many patches, the saliency signal becomes noisy and the reported gains would shrink.

Editorial extensions

If this is right

  • Saliency-guided masking alone nearly doubles MIMIC-CXR micro CE F1 over random masking in the paper's ablation (0.294 to 0.499), so the localization step, not just the MAE reconstruction, drives the clinical gain.
  • Saliency-guided decoding is a smaller but real contributor (0.294 to 0.380), implying that preserving sparse symptom clues during generation requires guidance at the text side as well.
  • Because the saliency map is learned from image-text pairs rather than from region annotations, the method can be applied to any dataset that already has reports.
  • The reported MIMIC-CXR numbers exceed all listed specialized baselines and several medical LLMs, suggesting that focused task-specific models can still beat generalist medical foundation models on narrow RRG tasks.
  • A masking-probability increment around $\varphi=0.35$ is optimal in the paper's ablation; too little makes the guidance negligible and too much destroys the features needed for reconstruction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the saliency map's reliance on a max-pooled global feature may be sensitive to dataset priors; on datasets where large normal structures dominate, top-k patches could drift toward frequent anatomy, so the optimal $\varphi$ and top-k fraction likely need per-dataset tuning.
  • Editorial inference: the same saliency-guided masking recipe could transfer to other sparse-finding medical modalities such as ultrasound or CT if a text encoder aligned to those reports is available, but the contrastive alignment would need to be retrained on matched image-report pairs.
  • Editorial inference: because the saliency map is trained on the same dataset's reports, it may inherit dataset-specific annotation and prevalence biases; evaluating CheXbert F1 on reports from an external hospital or on an artificially prevalence-shifted test set would test how portable the gains are.
  • Editorial inference: the comparison against medical LLMs uses the evaluation protocol of one of those models (MedDr), so a head-to-head run under identical split, beam size, and prompt settings would clarify how much of the margin is method versus protocol.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes SISRNet, a radiology report generation method for chest X-rays. The key idea is to identify "semantically informed salient regions" via a contrastively trained cross-modal space: a saliency map is computed as the cosine similarity between each image patch embedding and the max-pooled global image embedding (Eq. 3). These salient regions are then used in two ways: to bias masked image modeling so that the encoder focuses on pathological areas (Eq. 4 and Eq. 5), and to inject a saliency-weighted visual token into the report decoder (Eq. 6). The authors evaluate on IU-Xray and MIMIC-CXR, comparing against ten prior RRG methods and five medical LLMs, and report consistent improvements in NLG and clinical-efficacy metrics, with ablations on MIMIC-CXR.

Significance. If the reported results are robust, SISRNet would be a strong new state of the art on two standard RRG benchmarks, with a conceptually simple mechanism for focusing on disease-relevant regions. The paper has clear strengths: it compares with a wide range of baselines, reports clinical-efficacy metrics in addition to n-gram metrics, and provides component ablations showing that both the salient-region masking and the saliency-guided language model contribute to the gains. However, the quantitative claims are currently supported only by single-run numbers without error bars or significance tests, the central saliency mechanism lacks a non-semantic control, and the hyperparameter φ is selected on the test set. The significance is therefore conditional on addressing these concerns.

major comments (6)
  1. [Section IV-B and Table I] The text claims that SISRNet 'achieves superior performance across all three CE metrics,' but Table I lists no CE values for IU-Xray (all entries are '- - -'). Thus the clinical-efficacy claim is empirically supported only on MIMIC-CXR. The abstract and conclusion should either be qualified or the IU-Xray CE results should be supplied.
  2. [Section III-A, Eq. (8)] The identification network's training loss is defined as L1 = λvLv + λtLt + λbiLbi, but Lv and Lt are never defined anywhere in the paper; only Lv→t and Lt→v from Eq. (2) are specified. Since the contrastively learned common semantic space is the basis of the saliency map in Eq. (3), this omission prevents the reader from verifying what objective actually shapes the salient-region identification network. The authors should define Lv and Lt or remove them from Eq. (8).
  3. [Section III-A and Table III] The component ablations in Table III show that both SISR-Masking and SISR-LM improve over BASE, but both components consume the same semantically informed saliency map. There is no control where the cross-modal identification network is replaced by an image-only, non-semantic saliency baseline (e.g., cosine similarity between raw ViT patch features and a global feature, or random top-k patch selection). Without such a control, the paper's central claim that 'fine-grained cross-modal semantics' is the source of the gains is not isolated; the improvements could equally come from any patch-selection bias. Please add this control.
  4. [Section IV-A, Table IV] The masking-probability increment φ is selected by evaluating on the MIMIC-CXR test set (Table IV) and then the final reported result uses the best test-set value (φ = 0.35). No validation-based selection is described. In addition, no error bars or significance tests are reported for any table, so the reader cannot assess whether the reported differences, including the large F1 gap of 0.509 vs. 0.391, are stable across runs. This test-set tuning combined with missing variance may inflate the claimed improvements.
  5. [Section II-B, Table II] The comparison with medical LLMs in Table II is not adequately controlled. SISRNet is fine-tuned on MIMIC-CXR, while the results for RadFM, LLaVA-Med, Med-Flamingo, InternVL, and MedDr appear to be taken from prior work, which may use different evaluation protocols or may not be fine-tuned for this specific report generation task. The paper should state the exact adaptation protocol for each compared model and, ideally, evaluate all methods under the same setting to make the comparison fair.
  6. [Section III-B and Table III] The BASE model used in Table III is not sufficiently described: its encoder-decoder architecture, pretraining, and training details are not specified. Likewise, the masking procedure from Eq. (4) does not explain how the probability increments are converted to the announced 75% mask ratio, nor whether the updated probabilities are clipped or renormalized. These details are necessary for reproducibility and for interpreting the ablation's baseline.
minor comments (6)
  1. [Throughout] There are multiple typos and spelling errors, including 'visiual' (Section II-A), 'knowledege' (Section II-A), 'auxilliary' (Sections I and II-A), 'piroritizes' (Section II-B), 'corss-modal' (Section III-A), 'datesets' (Section IV-A), 'discriptions' (Section III-B), and 'Concluesion' (Section V). A careful proofread is needed.
  2. [Eq. (2) and Eq. (8)] The notation λv and λt is used for two different purposes: in Eq. (2) they weight the two contrastive directions, while in Eq. (8) they weight Lv and Lt. This is confusing and should be changed to distinct symbols.
  3. [Section III-D and Section IV-A] The weight names are inconsistent: the text uses λv→t and λt→v in experiments, while Eq. (2) uses λv and λt; the paper also does not define all loss weights in one place. Please unify the notation and list all hyperparameter values with their roles.
  4. [Section IV-A] The phrase 'We set the masking rate to 75% in report generation network' is unclear: it is likely intended to mean the language generation network, but the masking rate is also relevant to masked image modeling. Please clarify which masking is being set to 75%.
  5. [Table II] Table II reports only NLG metrics (BLEU, METEOR, ROUGE-L) and omits clinical-efficacy metrics, which the paper itself argues are the more important evaluation for RRG. Adding CE metrics would make the LLM comparison more informative.
  6. [References] Reference [30] is cited as an arXiv preprint; if a peer-reviewed version now exists, the authors should update the citation. Additionally, several related works are discussed without a direct citation in the text (e.g., the 'masking strategies' paragraph in Section II-B cites [19], [20], [27]-[29] but not all are discussed explicitly).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the saliency map is an intermediate image-derived representation, and the only self-citation (MbA, [30]) is not load-bearing.

full rationale

The paper's derivation chain is not circular. The saliency map in Eq. (3) is computed as cosine similarity between local ViT patch features and the max-pooled global image feature in a contrastively learned space. This is an intermediate representation derived from the input image, not a fitted parameter renamed as a prediction. The saliency map is then used in Eq. (4) to bias masked image modeling and in Eq. (6) to form an extra language-model token; neither step presumes the output report. The final report is generated from image features and the saliency-weighted token, and the training loss is cross-entropy against ground-truth reports, so the reported test-set results are genuine held-out evaluations. The only self-reference is the 'Mapping before Aggregate' (MbA) strategy from the authors' prior work [30], used in Section III-A to construct the common semantic space. This citation is not load-bearing: the strategy is fully described in the present paper, no uniqueness or external-support claim rests on it, and the central contribution is validated by external benchmarks (IU-Xray, MIMIC-CXR) and ablations. Consequently, there is no circular step by construction, and the paper warrants a low score reflecting only a minor, non-load-bearing self-citation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The performance of SISRNet rests on a set of hyperparameters (phi, top k%, masking rate, loss weights, temperature) that are set by hand or by ablation, and on the untested modeling assumption that cosine similarity between local patches and a max-pooled global feature in a contrastive space marks medically salient regions. No code or formal proof is provided for any component, so the reader must accept these assumptions without independent verification.

free parameters (5)
  • phi = 0.35
    Masking probability increment for salient regions in Eq. (4); chosen by ablation on MIMIC-CXR (Table IV), tuning on the test split without a reported validation split.
  • top k% = 20%
    Proportion of local features selected as salient regions (Section IV-A); set without reported ablation or sensitivity analysis.
  • masking rate = 75%
    Fraction of image tokens masked in the MAE objective (Section IV-A); chosen by hand.
  • loss weights = lambda_v_to_t=0.75, lambda_t_to_v=0.25, lambda_v=1.0, lambda_t=1.0, lambda_bi=0.1, lambda_I=1.0, lambda_R=0.1
    Weights in Eqs. (8) and (9); set by hand or validation, no sensitivity analysis except for phi.
  • temperature tau = not reported
    Temperature in the contrastive loss Eq. (2); value is not specified, needed for reproduction.
assumptions (4)
  • domain assumption The common semantic space learned by the bidirectional contrastive loss aligns local image patches with report tokens such that similarity to the global feature indicates medical salience.
    Section III-A, Eq. (3). The entire saliency map rests on this untested modeling premise.
  • domain assumption Masking salient regions during masked image modeling forces the encoder to learn refined representations of subtle abnormalities.
    Section III-B; supported only by ablations on one dataset, no theoretical or independent evidence.
  • domain assumption CheXbert-extracted labels and micro-averaged CE metrics measure the clinical accuracy of generated reports.
    Section IV-A; CheXbert is an automated labeler, not ground truth from radiologists, so the clinical accuracy claim inherits the labeler's limitations.
  • domain assumption Pre-trained encoders (ViT-B16, BioClinicalBERT) provide features that preserve the medical semantics needed for alignment.
    Section III-A; the method assumes these off-the-shelf features are sufficient for fine-grained cross-modal alignment in chest X-rays.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantically Informed Salient Regions Guided Radiology Report Generation." pith.science (2026). https://pith.science/paper/E4GR4XW3

@misc{pith2026250711015,
  author       = {Pith},
  title        = {Pith review of: Semantically Informed Salient Regions Guided Radiology Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E4GR4XW3}},
  note         = {Machine review of arXiv:2507.11015}
}
read the original abstract

Recent advances in automated radiology report generation from chest X-rays using deep learning algorithms have the potential to significantly reduce the arduous workload of radiologists. However, due to the inherent massive data bias in radiology images, where abnormalities are typically subtle and sparsely distributed, existing methods often produce fluent yet medically inaccurate reports, limiting their applicability in clinical practice. To address this issue effectively, we propose a Semantically Informed Salient Regions-guided (SISRNet) report generation method. Specifically, our approach explicitly identifies salient regions with medically critical characteristics using fine-grained cross-modal semantics. Then, SISRNet systematically focuses on these high-information regions during both image modeling and report generation, effectively capturing subtle abnormal findings, mitigating the negative impact of data bias, and ultimately generating clinically accurate reports. Compared to its peers, SISRNet demonstrates superior performance on widely used IU-Xray and MIMIC-CXR datasets.

Figures

Figures reproduced from arXiv: 2507.11015 by the authors.

Figure 1
Figure 1. Overview of the proposed radiology report generation method. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Salient regions identification network. X-rays, they provide limited assistance for generating long reports with complex and scattered semantics. In contrast, our method identifies semantically informed salient regions and piroritizes these high-information regions in both image modeling and report generation, detecting subtle abnormalities and generating clinically accurate reports. III. APPROACH The overall archit… view at source ↗
Figure 3
Figure 3. Case of reports generated by SISRNet on the MIMIC-CXR [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 44 canonical work pages

  1. [1]

    Review on covid-19 diagnosis models based on machine learning and deep learning approaches,

    Z. A. A. Alyasseri, M. A. Al-Betar, I. A. Doush, M. A. Awadallah, A. K. Abasi, S. N. Makhadmeh, O. A. Alomari, K. H. Abdulkareem, A. Adam, R. Damasevicius, et al. , “Review on covid-19 diagnosis models based on machine learning and deep learning approaches,” Expert systems , vol. 39, no. 3, p. e12759, 2022

    work page 2022

  2. [2]

    Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images,

    T. Rahman, A. Khandakar, Y . Qiblawey, A. Tahir, S. Kiranyaz, S. B. A. Kashem, M. T. Islam, S. Al Maadeed, S. M. Zughaier, M. S. Khan,et al., “Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images,” Computers in biology and medicine, vol. 132, p. 104319, 2021

    work page 2021

  3. [3]

    Show and tell: A neural image caption generator,

    O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3156–3164, 2015

    work page 2015

  4. [4]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hocken- maier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proceedings of the IEEE international conference on computer vision , pp. 2641–2649, 2015

    work page 2015

  5. [5]

    Cross-modal contrastive attention model for medical report generation,

    X. Song, X. Zhang, J. Ji, Y . Liu, and P. Wei, “Cross-modal contrastive attention model for medical report generation,” in Proceedings of the 29th International Conference on Computational Linguistics , pp. 2388– 2397, 2022

    work page 2022

  6. [6]

    Translating medical im- age to radiological report: Adaptive multilevel multi-attention approach,

    G. O. Gajbhiye, A. V . Nandedkar, and I. Faye, “Translating medical im- age to radiological report: Adaptive multilevel multi-attention approach,” Computer Methods and Programs in Biomedicine , vol. 221, p. 106853, 2022

    work page 2022

  7. [7]

    Automatic medical report generation based on cross-view attention and visual-semantic long short term memorys,

    Y . Gu, R. Li, X. Wang, and Z. Zhou, “Automatic medical report generation based on cross-view attention and visual-semantic long short term memorys,” Bioengineering, vol. 10, no. 8, p. 966, 2023

    work page 2023

  8. [8]

    Improving factual completeness and consistency of image-to-text radiology report generation,

    Y . Miura, Y . Zhang, E. Tsai, C. Langlotz, and D. Jurafsky, “Improving factual completeness and consistency of image-to-text radiology report generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics , pp. 5288–5304, 2021

    work page 2021

Show all 50 references
  1. [9]

    Improving the factual correctness of radiology report generation with semantic rewards,

    J.-B. Delbrouck, P. Chambon, C. Bluethgen, E. Tsai, O. Almusa, and C. Langlotz, “Improving the factual correctness of radiology report generation with semantic rewards,” in Findings of the Association for Computational Linguistics, pp. 4348–4360, 2022

    2022

  2. [10]

    Dynamic graph enhanced contrastive learning for chest x-ray report generation,

    M. Li, B. Lin, Z. Chen, H. Lin, X. Liang, and X. Chang, “Dynamic graph enhanced contrastive learning for chest x-ray report generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3334–3343, 2023

    2023

  3. [11]

    Attributed abnormality graph embedding for clinically accurate x-ray report generation,

    S. Yan, W. K. Cheung, K. Chiu, T. M. Tong, K. C. Cheung, and S. See, “Attributed abnormality graph embedding for clinically accurate x-ray report generation,” IEEE Transactions on Medical Imaging , vol. 42, no. 8, pp. 2211–2222, 2023

    2023

  4. [12]

    Embracing uniqueness: Gen- erating radiology reports via a transformer with graph-based distinctive attention,

    H. Wang, J. Niu, X. Liu, and Y . Wang, “Embracing uniqueness: Gen- erating radiology reports via a transformer with graph-based distinctive attention,” in 2022 IEEE International Conference on Bioinformatics and Biomedicine, pp. 581–588, 2022

    2022

  5. [13]

    Clinically accurate chest x-ray report generation,

    G. Liu, T.-M. H. Hsu, M. McDermott, W. Boag, W.-H. Weng, P. Szolovits, and M. Ghassemi, “Clinically accurate chest x-ray report generation,” in Machine Learning for Healthcare Conference , pp. 249– 269, PMLR, 2019

    2019

  6. [14]

    Automated generation of accurate & fluent medical x-ray reports,

    H. Nguyen, D. Nie, T. Badamdorj, Y . Liu, Y . Zhu, J. Truong, and L. Cheng, “Automated generation of accurate & fluent medical x-ray reports,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 3552–3569, 2021

    2021

  7. [15]

    Organ: Observation-guided radiology report generation via tree reasoning,

    W. Hou, K. Xu, Y . Cheng, W. Li, and J. Liu, “Organ: Observation-guided radiology report generation via tree reasoning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8108–8122, 2023

    2023

  8. [16]

    Radiology report generation with a learned knowledge base and multi-modal alignment,

    S. Yang, X. Wu, S. Ge, Z. Zheng, S. K. Zhou, and L. Xiao, “Radiology report generation with a learned knowledge base and multi-modal alignment,” Medical Image Analysis , vol. 86, p. 102798, 2023

    2023

  9. [17]

    Multi-grained radiology report generation with sentence-level image-language contrastive learning,

    A. Liu, Y . Guo, J.-h. Yong, and F. Xu, “Multi-grained radiology report generation with sentence-level image-language contrastive learning,” IEEE Transactions on Medical Imaging , 2024

    2024

  10. [18]

    Camanet: class activation map guided attention network for radiology report generation,

    J. Wang, A. Bhalerao, T. Yin, S. See, and Y . He, “Camanet: class activation map guided attention network for radiology report generation,” IEEE Journal of Biomedical and Health Informatics , 2024

    2024

  11. [19]

    Focusmae: Gallbladder cancer detection from ultrasound videos with focused masked autoencoders,

    S. Basu, M. Gupta, C. Madan, P. Gupta, and C. Arora, “Focusmae: Gallbladder cancer detection from ultrasound videos with focused masked autoencoders,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11715–11725, 2024

    2024

  12. [20]

    Rethinking masked image modelling for medical image representation,

    Y . Xie, L. Gu, T. Harada, J. Zhang, Y . Xia, and Q. Wu, “Rethinking masked image modelling for medical image representation,” Medical Image Analysis, vol. 98, p. 103304, 2024

    2024

  13. [21]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation,

    K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in Conference on Empirical Methods in Natural Language Processing , 2014

    2014

  14. [22]

    Sequence to sequence learning with neural networks,

    I. Sutskever, O. Vinyals, and Q. V . Le, “Sequence to sequence learning with neural networks,” in Proceedings of the 28th International Confer- ence on Neural Information Processing Systems , pp. 3104–3112, 2014

    2014

  15. [23]

    Cross-modal memory networks for radiology report generation,

    Z. Chen, Y . Shen, Y . Song, and X. Wan, “Cross-modal memory networks for radiology report generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume...

    2021

  16. [24]

    Matnet: Exploiting multi-modal features for radiology report generation,

    C. Shang, S. Cui, T. Li, X. Wang, Y . Li, and J. Jiang, “Matnet: Exploiting multi-modal features for radiology report generation,” IEEE Signal Processing Letters , vol. 29, pp. 2692–2696, 2022

    2022

  17. [25]

    Semi-supervised medical report generation via graph-guided hybrid fea- ture consistency,

    K. Zhang, H. Jiang, J. Zhang, Q. Huang, J. Fan, J. Yu, and W. Han, “Semi-supervised medical report generation via graph-guided hybrid fea- ture consistency,” IEEE Transactions on Multimedia , vol. 26, pp. 904– 915, 2023

    2023

  18. [26]

    Visual-textual cross- modal interaction network for radiology report generation,

    W. Zhang, B. Cai, J. Hu, Q. Qin, and K. Xie, “Visual-textual cross- modal interaction network for radiology report generation,” IEEE Signal Processing Letters, 2024

    2024

  19. [27]

    Mgmae: Motion guided masking for video masked autoencoding,

    B. Huang, Z. Zhao, G. Zhang, Y . Qiao, and L. Wang, “Mgmae: Motion guided masking for video masked autoencoding,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 13493– 13504, 2023

    2023

  20. [28]

    Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders,

    W. G. C. Bandara, N. Patel, A. Gholami, M. Nikkhah, M. Agrawal, and V . M. Patel, “Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14507– 14517, 2023

    2023

  21. [29]

    Medim: Boost medical image representation via radiology report-guided masking,

    Y . Xie, L. Gu, T. Harada, J. Zhang, Y . Xia, and Q. Wu, “Medim: Boost medical image representation via radiology report-guided masking,” in International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 13–23, Springer, 2023. JOURNAL OF LATEX CLA...

    2023

  22. [30]

    Masked contrastive reconstruc- tion for cross-modal medical image-report retrieval,

    Z. Wei, K. Jin, and X. Zhou, “Masked contrastive reconstruc- tion for cross-modal medical image-report retrieval,” arXiv preprint arXiv:2312.15840, 2023

    2023 arXiv

  23. [31]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations , 2020

    2020

  24. [32]

    Publicly available clinical bert embeddings,

    E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng, D. Jindi, T. Naumann, and M. McDermott, “Publicly available clinical bert embeddings,” in Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72–78, 2019

    2019

  25. [33]

    Cross-modal prototype driven network for radiology report generation,

    J. Wang, A. Bhalerao, and Y . He, “Cross-modal prototype driven network for radiology report generation,” in European Conference on Computer Vision, pp. 563–579, 2022

    2022

  26. [34]

    Kiut: Knowledge-injected u- transformer for radiology report generation,

    Z. Huang, X. Zhang, and S. Zhang, “Kiut: Knowledge-injected u- transformer for radiology report generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 19809–19818, 2023

    2023

  27. [35]

    An organ-aware diagnosis framework for radiology report generation,

    S. Li, P. Qiao, L. Wang, M. Ning, L. Yuan, Y . Zheng, and J. Chen, “An organ-aware diagnosis framework for radiology report generation,” IEEE Transactions on Medical Imaging , 2024

    2024

  28. [36]

    Phraseaug: An augmented medical report generation model with phrasebook,

    X. Mei, L. Yang, D. Gao, X. Cai, J. Han, and T. Liu, “Phraseaug: An augmented medical report generation model with phrasebook,” IEEE Transactions on Medical Imaging , 2024

    2024

  29. [37]

    Preparing a collection of radiology examinations for distribution and retrieval,

    D. Demner-Fushman, M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez, S. Antani, G. R. Thoma, and C. J. McDonald, “Preparing a collection of radiology examinations for distribution and retrieval,” Journal of the American Medical Informatics Association, vol. 23, no. 2,...

    2016

  30. [38]

    Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports,

    A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng, “Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports,” Scientific data, vol. 6, no. 1, p. 317, 2019

    2019

  31. [39]

    Generating radiology reports via memory-driven transformer,

    Z. Chen, Y . Song, T.-H. Chang, and X. Wan, “Generating radiology reports via memory-driven transformer,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1439–1449, 2020

    2020

  32. [40]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pp. 311–318, 2002

    2002

  33. [41]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pp. 65–72, 2005

    2005

  34. [42]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , pp. 74–81, 2004

    2004

  35. [43]

    Baselines for chest x-ray report generation,

    W. Boag, T.-M. H. Hsu, M. McDermott, G. Berner, E. Alesentzer, and P. Szolovits, “Baselines for chest x-ray report generation,” in Machine learning for health workshop , pp. 126–140, PMLR, 2020

    2020

  36. [44]

    Interactive and ex- plainable region-guided radiology report generation,

    T. Tanida, P. M ¨uller, G. Kaissis, and D. Rueckert, “Interactive and ex- plainable region-guided radiology report generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7433–7442, 2023

    2023

  37. [45]

    Combining automatic labelers and expert annotations for accurate radiology report labeling using bert,

    A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y . Ng, and M. Lungren, “Combining automatic labelers and expert annotations for accurate radiology report labeling using bert,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp....

    2020

  38. [46]

    Towards generalist foundation model for radiology,

    C. Wu, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Towards generalist foundation model for radiology,” arXiv preprint arXiv:2308.02463 , 2023

    2023 arXiv

  39. [47]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” Advances in Neural Information Processing Systems, vol. 36, pp. 28541–28564, 2023

    2023

  40. [48]

    Med-flamingo: a multimodal medical few-shot learner,

    M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y . Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar, “Med-flamingo: a multimodal medical few-shot learner,” in Machine Learning for Health , pp. 353– 367, 2023

    2023

  41. [49]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. , “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2024

  42. [50]

    Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning,

    S. He, Y . Nie, Z. Chen, Z. Cai, H. Wang, S. Yang, and H. Chen, “Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning,” arXiv preprint arXiv:2404.15127 , 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.