Pith. sign in

REVIEW 3 major objections 6 minor 84 references

Automatic Fine-grained Segmentation-assisted Report Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding specialist segmentation maps and intermediate features to LLaVA's projector raises chest X-ray report CE F1 by 2.77 points over the two-stage baseline.

desk verdict Simple modular fusion method with a clean ablation, but the grounding claim is undercut by the paper's own shuffled-map control, and test-set selection makes the p-values optimistic. read the letter →

arxiv 2507.16623 v1 pith:ZSLTMI3G submitted 2025-07-22 cs.CV cs.LG

Frederic Jonske , Constantin Seibold , Osman Alperen Koras , Fin Bahnsen , Marie Bauer , Amin Dada , Hamza Kalisch , Anton Schily
show 1 more author
Jens Kleesiek
This is my paper
classification cs.CVcs.LG
keywords radiologyreportgenerationLLaVAextensionsegmentation-guidedvision-languagemodelclinicalefficacyscoreMIMIC-CXRgroundedCXASsegmentationmapsLVM-Medfeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ASaRG claims that a LLaVA-style report generator becomes both more clinically accurate and more verifiable if, alongside the chest X-ray, it is given two specialist inputs: intermediate features from a medical segmentation foundation model and 212 fine-grained anatomical, pathological, and foreign-object segmentation maps. The paper fuses these into LLaVA's multi-modal projector by simple concatenation and reports, on MIMIC-CXR, a +2.77 percentage-point gain in the 14-class clinical-efficacy F1 (CE F1) over a two-stage LLaVA baseline (p<0.001), at roughly 0.09% added parameters. It also reports outperforming two segmentation-based report generators, COMG and ORID, by 6.98 and 6.28 F1 points respectively. The broader claim is that the maps let a reader trace generated report statements to segmentation evidence, giving a natural route to grounded report verification. A sympathetic reader would care because the recipe is modular: it adds cheap, specialist local cues without retraining the vision tower or language backbone.

What carries the argument

ASaRG's load-bearing machinery is a concatenation-based modification of LLaVA's multi-modal projector. LLaVA normally computes $O=\mathrm{LLM}(\mathrm{CAT}(P(\mathbf{F}_I), \mathbf{F}_T))$; ASaRG instead computes $O=\mathrm{LLM}(\mathrm{CAT}(P^*(\mathbf{F}_I,\mathbf{F}_{\mathrm{new}}), \mathbf{F}_T))$, where $\mathbf{F}_{\mathrm{new}}$ is built from a learnable class embedding $\mathbf{C}$, LVM-Med intermediate features $\mathbf{R}_I$, and down-pooled segmentation maps $\mathbf{S}_I$. The auxiliary inputs are aligned with small linear layers (and adaptive pooling plus a 1D convolution for the maps), then concatenated along the channel and sequence axes with the original vision features $\mathbf{F}_I$ before the pretrained projector $P$ and a frozen LLM. Concatenation is the decisive choice: the original vision-to-language alignment is kept intact, and any number of additional specialist modalities can be appended with negligible parameter cost.

What would settle it

Take a fixed set of test images and replace the CXAS maps with maps from a segmentation model with substantially higher per-class pathology recall, or with radiologist-corrected maps; if CE F1 and the shuffled-map gap do not rise, the mechanism is not map accuracy. Alternatively, train ASaRG with the 54 new pathological and foreign-object classes removed: if CE F1 on the pathology-related classes is unchanged, the extension's central premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a LLaVA-based report generator can be improved by simply concatenating two additional visual modalities into its projector: intermediate embeddings from LVM-Med, a medical segmentation foundation model, and 212 full-size segmentation maps from an extended CXAS segmenter covering anatomy, pathologies, and foreign objects. In the ASaRG configuration, the new sequence of vision features passes through LLaVA's pretrained projector, so the language model can attend to specialist local cues without changing the pretrained weights. On MIMIC-CXR the authors report CE F1 rising from 0.3872 for the two-stage LLaVA baseline to 0.4149 (+2.77 points, p<0.001), beating a fully finetuned LLaVA by 0.94 points (p=0.007) despite freezing the vision tower and the LLM. They further state that because the segmentation maps are explicit inputs, elements of the generated report can be checked against the corresponding maps, demonstrably allowing tracing of assessments to segmentation evidence and verification of their groundedness.

Load-bearing premise

The load-bearing premise is that the offline CXAS segmentation maps, including the 54 newly added pathological and foreign-object classes, are accurate enough on MIMIC-CXR to deliver usable local cues; if the maps are too noisy, the reported gain is mostly extra model capacity or feature interaction, and the grounding claim is not demonstrated.

Editorial extensions

If this is right

  • Concatenation is the best of the four tested fusion strategies, beating learned mixing, weighted addition, and full replacement, while adding about 0.06% parameters in the one-epoch setting.
  • Adding the 212 segmentation maps on top of the intermediate features gives a further +1.88 percentage points of CE F1, for +2.77 points over the two-stage LLaVA baseline with roughly 0.09% added projector parameters.
  • ASaRG outperforms the two prior segmentation-guided report generators, COMG and ORID, by 6.98 and 6.28 CE F1 points, even though its lexical similarity scores are lower.
  • Because the method freezes the vision tower and LLM and only touches the projector, it is compatible with other LLaVA-based advances, such as better encoders or language backbones, without retraining from scratch.
  • The 20-superclass version matches the 212-class version within 0.01 CE F1, so the reported benefit appears insensitive to segmentation granularity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the shuffled-map control implies the map-specific grounding contribution is only about +0.33 of the +2.77 points; the remainder likely comes from feature-map interaction or extra projection capacity, so future work should separate these before crediting the maps.
  • Editorial inference: because 212 classes and 20 superclasses perform identically, investing in cheaper, cleaner segmentation (fewer but more accurate classes) may capture most of the benefit while reducing compute and memory.
  • Editorial inference: the authors' own figure shows the segmenter missing atelectasis and pleural effusions; improving pathology recall in the segmenter, not just adding more classes, should convert part of the capacity-driven gain into genuinely grounded gain, which is testable by swapping in a higher-recall segmenter and checking whether the shuffled-map gap grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents ASaRG, an extension of LLaVA for chest X-ray report generation. In addition to the standard LLaVA image features, ASaRG concatenates intermediate embeddings from the LVM-Med segmentation foundation model and 212-class segmentation maps from a CXAS-based segmenter into the LLaVA multimodal projector, using a learnable class embedding and lightweight linear layers. The method is evaluated on a VQA-formatted version of MIMIC-CXR, with four repeated runs per configuration and Welch's t-tests on the Clinical Efficacy F1 score. The authors report a +0.89 percentage-point gain when adding intermediate features alone and a +2.77 percentage-point gain when adding features plus fine-grained segmentation maps over a two-stage LLaVA baseline, with only about 0.09% added parameters. They also claim that the segmentation maps enable tracing elements of the generated report to the corresponding maps, and present a shuffled-map control in Section 6.1 to assess how much of the gain is due to the specific map content.

Significance. If the reported performance gain and the grounding capability were both substantiated, ASaRG would be a valuable lightweight contribution: it injects specialist knowledge into a LLaVA-style model without fine-tuning the backbone, maintains most of the original architecture, and opens a route toward visually grounded report generation. The paper has notable strengths in experimental practice: four repetitions per configuration, explicit p-values, a shuffled-map control that is uncommon and informative, and a careful accounting of parameter overhead. However, the shuffled-map control in Section 6.1 shows that only about +0.33 percentage points of the headline +2.77 gain survive randomizing the class-to-map correspondence, meaning the paper's central claim about the specific content of the fine-grained segmentation maps is not strongly supported. The grounding claim in the abstract and Section 6.2 is correspondingly weakened. The finding that a simple concatenation of extra features yields a stable CE F1 improvement is still of interest, but the stated contribution of the 212-class fine-grained maps needs better evidence.

major comments (3)
  1. [Section 6.1, Table 2] The shuffled-map control shows that randomizing the order of the segmentation maps on the test set removes only about +0.33 percentage points from the +2.77 percentage-point gain over the two-stage LLaVA baseline. This leaves about +2.44 percentage points of improvement under shuffled maps, which the authors attribute to projection capacity, features-map interaction, or generic map statistics. Given this result, the abstract's and Section 6.2's claim that the method 'demonstrably allows tracing elements of the report to the corresponding segmentation maps and verifying the groundedness of assessments' is not supported: if destroying the class-to-map correspondence leaves most of the gain intact, the generator is not shown to use the local, class-specific cues that grounding requires. Please either provide a direct grounding evaluation (for example, human raters judging whether report phrases correspond to the correct segmentation maps in sorted versus shuffled conditions, or an automatic per-class localization metric), or substantially temper the grounding claim to describe the architecture as enabling manual verification of correspondences rather than demonstrating that the model's outputs are actually grounded.
  2. [Section 4.1.1 and Tables 1-2] The concatenation fusion method was selected by comparing four fusion variants on the same holdout test set that is later used to compute the p-values against the LLaVA baselines. Because the test set was used for model selection, the reported p<0.001 and p=0.007 are selection-optimistic: the test statistic is effectively the maximum over the four configurations, so the probability of observing at least one significant improvement by chance is larger than the nominal level. The effect size is large and may well survive a proper treatment, but the p-values as reported are not statistically valid. Please either reserve a separate validation set for fusion selection and evaluate only the chosen configuration on the test set, or apply a multiple-comparison correction and state that the p-values are post-selection.
  3. [Section 6.2 and Figure 2] The two qualitative grounding examples are the only direct evidence for the grounding claim, and the second example explicitly shows the segmentation model missing atelectasis and pleural effusions that appear in the generated report. This is a false negative in the proposed verification procedure for two of the most clinically important findings. One or two curated examples, one of which fails on central pathologies, cannot support the general claim that ASaRG demonstrably allows verifying the groundedness of assessments. A systematic analysis is needed: for a random sample of test-set reports, how often do generated pathological findings have corresponding segmentation maps, and how does that correspondence rate compare on sorted versus shuffled maps? Without such evaluation, the grounding assertion should be removed or reduced to a suggestion for future work.
minor comments (6)
  1. [Section 3.3, Eq. (4)-(9)] The notation for the fusion functions is inconsistent: Eq. (4) defines Fnew = f(C,R,S), but the text then denotes the features after mixing as RI and the segmentation-derived features as SI, and f and g are used without precise signatures. Please state explicitly the dimensions of C, Rstack, and the inputs to Lin1, Lin2, and the segmentation branch, so the described variants can be reimplemented without ambiguity.
  2. [Abstract and Table 2] The abstract reports '+0.89% performance gain' for the features-only condition, while Table 2 shows +0.89 percentage points (0.3961 vs 0.3872) and Table 1 shows +0.87 percentage points for the one-epoch features-only comparison. Please use percentage points consistently and indicate which table corresponds to the abstract value.
  3. [Section 6.1 and Table 2] The shuffled-map result is reported as a difference of about +0.33% without a standard error or p-value. Since each run was repeated four times, please report the mean and standard deviation of the sorted-minus-shuffled difference and state whether +0.33% is statistically significant across the runs. This is important because the paper interprets this small difference as the entire interpretable contribution of the segmentation maps.
  4. [Section 4.1.2 and Supplementary Materials] The two-stage training recipe differs between the features-only and features-plus-segmentation conditions not only in the presence of segmentation maps but also in the number of trainable stages and the parameter count. The paper argues convincingly in Section 6.1 that parameter count alone cannot explain the full gain, but the comparisons in Table 2 would be cleaner if the two-stage features-only condition and the two-stage features-plus-maps condition had identical training schedules except for the added branch. Please state explicitly whether this is the case.
  5. [Reproducibility Statement] The code is promised 'at a later date' and the CXAS additional-class training data are not included. For the field to build on the 212-class segmentation setup, the extended CXAS class definitions and the finetuning data (or a description sufficient to reproduce them) should be released alongside the paper.
  6. [Section 6.3 and Supplementary Figures 4-5] The supplementary material reports that the per-class advantage of ASaRG is diffuse and varies across repeated runs, and that differences on some classes could not be determined due to class imbalance. Please mention this in the main text's limitations section, since it qualifies the clinical-efficacy interpretation of the aggregate CE F1 improvement.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline CE F1 gain rests on a held-out empirical comparison, not on a self-referential derivation; the shuffled-map control directly probes map contribution.

full rationale

The central claim—that concatenating LVM-Med intermediate features and CXAS segmentation maps into LLaVA's projector improves CE F1 on MIMIC-CXR—is an empirical finetuning result evaluated on the holdout test set. The report generator is trained on reference reports, not on the segmentation model's outputs, so the reported metric is not a by-construction match to the extra inputs. The only self-citation is CXAS [36], co-developed by two present authors, used to generate the input maps; the maps' contribution is not asserted by citation but measured via the shuffled-map control in Section 6.1, which shows only about +0.33% of the gain depends on map order/content. That control is the correct empirical probe and its outcome weakens the paper's grounding interpretation, but a weak or negative control result is not circular reasoning. The paper also candidly lists residual explanations (projection capacity, features–map interaction, generic map statistics). A separate statistical concern is that the concatenation fusion was chosen after evaluating on the same holdout test set, making the p-values selection-optimistic; this is a validity limitation, not a definitional reduction. No equation in the paper reduces the claimed gain to its inputs by construction, and no uniqueness or ansatz result is imported from the authors' prior work.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality and transferability of offline segmentation and feature extraction, the validity of the VQA conversion and CheXbert scoring, and the assumption that concatenation does not destroy LLaVA's alignment. No fitted physical constants or derived formulas appear; the free parameters listed are hand-chosen experimental settings and learned network weights.

free parameters (3)
  • Number of finetuning epochs = 1 or 2
    All results use 1-2 epochs, a hand-chosen limit to reduce compute. The paper claims performance gaps remain stable but does not show saturation curves.
  • Segmentation class granularity = 212 classes or 20 superclasses
    The method uses 212 fine-grained classes; the superclass ablation performs identically (p=0.989), so this choice does not drive the result.
  • Learnable class embedding dimensions = width b*256, depth 512
    The projector's class embedding C uses these hand-selected dimensions, which set the added parameter count but are not fitted to a target value.
assumptions (4)
  • domain assumption MIMIC-CXR, converted to a VQA format with 33 paraphrased prompts and GPT-3.5-generated multi-turn chats, preserves the clinical content of the original findings reports.
    Section 3.1 builds the entire training and evaluation set this way. If the conversion changes the task, the CE F1 numbers do not reflect clinical report generation.
  • domain assumption CheXbert-based Clinical Efficacy F1 is a valid proxy for report quality.
    Section 4.2 relies on CE F1 as the primary metric. CheXbert's 14-class labeling of generated text is treated as ground truth, but its accuracy on synthetic reports is not validated here.
  • domain assumption Concatenating foreign embeddings into LLaVA's projector preserves the pretrained alignment well enough for 1-2 epochs of fine-tuning to be meaningful.
    The modified projector (Eq. 12) assumes the original CLIP-aligned vision tokens remain usable alongside new RI and SI tokens; the ablations support this empirically but it is structurally assumed.
  • domain assumption The offline LVM-Med and CXAS models produce informative inputs for MIMIC-CXR images, including the 54 newly added CXAS classes.
    Section 3.1 extracts features and maps offline. The 54 added classes are trained on ChestX-Det and CLiP, not MIMIC-CXR, so their transferability is assumed; Fig. 2 shows a failure case.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Fine-grained Segmentation-assisted Report Generation." pith.science (2026). https://pith.science/paper/ZSLTMI3G

@misc{pith2026250716623,
  author       = {Pith},
  title        = {Pith review of: Automatic Fine-grained Segmentation-assisted Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZSLTMI3G}},
  note         = {Machine review of arXiv:2507.16623}
}
abstract

Reliable end-to-end clinical report generation has been a longstanding goal of medical ML research. The end goal for this process is to alleviate radiologists' workloads and provide second opinions to clinicians or patients. Thus, a necessary prerequisite for report generation models is a strong general performance and some type of innate grounding capability, to convince clinicians or patients of the veracity of the generated reports. In this paper, we present ASaRG (\textbf{A}utomatic \textbf{S}egmentation-\textbf{a}ssisted \textbf{R}eport \textbf{G}eneration), an extension of the popular LLaVA architecture that aims to tackle both of these problems. ASaRG proposes to fuse intermediate features and fine-grained segmentation maps created by specialist radiological models into LLaVA's multi-modal projection layer via simple concatenation. With a small number of added parameters, our approach achieves a +0.89\% performance gain ($p=0.012$) in CE F1 score compared to the LLaVA baseline when using only intermediate features, and +2.77\% performance gain ($p<0.001$) when adding a combination of intermediate features and fine-grained segmentation maps. Compared with COMG and ORID, two other report generation methods that utilize segmentations, the performance gain amounts to 6.98\% and 6.28\% in F1 score, respectively. ASaRG is not mutually exclusive with other changes made to the LLaVA architecture, potentially allowing our method to be combined with other advances in the field. Finally, the use of an arbitrary number of segmentations as part of the input demonstrably allows tracing elements of the report to the corresponding segmentation maps and verifying the groundedness of assessments. Our code will be made publicly available at a later date.

Figures

Figures reproduced from arXiv: 2507.16623 by the authors.

Figure 1
Figure 1. The ASaRG architecture - Model elements of ASaRG are highlighted in green and different input modalities are high￾lighted in blue. Plus symbols denote concatenation operations. Italics in any component denote that the component is part of orig￾inal LLaVA architecture. 1. Introduction In recent years, multi-modal radiological report generation has made significant strides [45], both in terms of perfor￾mance and suppo… view at source ↗
Figure 2
Figure 2. Grounding with segmentation maps - This graphic depicts two examples from the MIMIC-CXR test set with ground truths and generated reports. The colors highlight parts of the gen￾erated report and their corresponding segmentation maps. Corre￾spondences are limited to a small amount of example classes, and reports are truncated to relevant sections for readability. worthy in this respect. In the second example, atelect… view at source ↗
Figure 3
Figure 3. The user queries u F i ∈ PF for prompting a VLM for radiology findings on an image vi, each paired with the findings r F i as the response to form a single-turn chat. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Per-class performance - This series of plots shows the per-class performance of different variants of ASaRG that use additional intermediate features, as well as the accompanying baselines [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Per-class performance - This series of plots shows the per-class performance of different variants of ASaRG that use additional intermediate features and (fine-grained) segmentation maps, as well as the accompanying baselines. 14 [PITH_FULL_IMAGE:figures/full_fig_p014…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 68 canonical work pages

  1. [1]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 6077–6086, 2018. 2

    2018

  2. [2]

    Qwen Technical Report, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  3. [3]

    Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, An- ton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Max- imilian Ilse, Fernando P ´erez-Garc´ıa, Valentina Salvatelli, 8 Harshita Sharma, Felix Meissen, Mercy Ranjit, Shaury Srivastav, Julia Gong, Noel C. F. Codella, Fabian Falck, Ozan Oktay, Matthew P. Lungren, Maria Teodora Wetscherek, Javier Alvarez-Valle...

    arXiv 2024

  4. [4]

    Baselines for Chest X-Ray Report Generation

    William Boag, Tzu-Ming Harry Hsu, Matthew Mcdermott, Gabriela Berner, Emily Alesentzer, and Peter Szolovits. Baselines for Chest X-Ray Report Generation. In Proceed- ings of the Machine Learning for Health NeurIPS Workshop, pages 126–140. PMLR, 2020. 5

    2020

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    1901

  6. [6]

    ImageNet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical im- age database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, 2009. IEEE. 12

    2009

  7. [7]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021. arXiv:2010.11929 [cs]. 3, 4

    arXiv 2021

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur ´elien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi `ere, B...

    arXiv 2024

Show all 84 references
  1. [9]

    PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physio- logic signals

    A L Goldberger, L A Amaral, L Glass, J M Hausdorff, P C Ivanov, R G Mark, J E Mietus, G B Moody, C K Peng, and H E Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physio- logic signals. Circulation, 101(23):E215–20, 2000. 3

    2000

  2. [10]

    Complex Organ Mask Guided Radiology Report Generation

    Tiancheng Gu, Dongnan Liu, Zhiyuan Li, and Weidong Cai. Complex Organ Mask Guided Radiology Report Generation. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7980–7989, Waikoloa, HI, USA, 2024. IEEE. 2, 3, 5, 6

    2024

  3. [11]

    ORID: Organ-Regional In- formation Driven Framework for Radiology Report Genera- tion

    Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Lin, and Weidong Cai. ORID: Organ-Regional In- formation Driven Framework for Radiology Report Genera- tion. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 378–387, Tucson, AZ, US...

    2025

  4. [12]

    Regiongpt: Towards region understanding vision lan- guage model

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision lan- guage model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13796– 1380...

    2024

  5. [13]

    CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging

    Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging . In proceedings of Medical Image Com- puting and Computer Assisted Intervention – MICCAI 2024. Springer Nature Switzerland, 2024. 1

    2024

  6. [14]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pages 272–284. Springer, 2021. 4

    2021

  7. [15]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 1501–1510, 2017. 4

    2017

  8. [16]

    Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C

    Stephanie L. Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Mercy Ranjit, Anton Schwaighofer, Fer- nando P ´erez-Garc´ıa, Valentina Salvatelli, Shaury Srivas- tav, Anja Thieme, Noel Codella, Matthew P. Lungren, Maria Teodora Wetscherek, Ozan Oktay, and Javier Alvarez-...

    2024 arXiv

  9. [17]

    Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radio- graphs with free-text reports. Sci Data, 6(1), 2019. P...

    2019

  10. [18]

    Why does 9 my medical AI look at pictures of birds? Exploring the ef- ficacy of transfer learning across domain boundaries

    Frederic Jonske, Moon Kim, Enrico Nasca, Janis Evers, Jo- hannes Haubold, Ren ´e Hosch, Felix Nensa, Michael Kamp, Constantin Seibold, Jan Egger, and Jens Kleesiek. Why does 9 my medical AI look at pictures of birds? Exploring the ef- ficacy of transfer learning across domain ...

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2017. arXiv:1412.6980 [cs]. 12

    2017 arXiv

  12. [20]

    Meteor: an automatic met- ric for MT evaluation with high levels of correlation with human judgments

    Alon Lavie and Abhaya Agarwal. Meteor: an automatic met- ric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation , pages 228–231, USA,

  13. [21]

    AutoRG-Brain: Grounded Report Generation for Brain MRI, 2024

    Jiayu Lei, Xiaoman Zhang, Chaoyi Wu, Lisong Dai, Ya Zhang, Yanyong Zhang, Yanfeng Wang, Weidi Xie, and Yue- hua Li. AutoRG-Brain: Grounded Report Generation for Brain MRI, 2024. arXiv:2407.16684 [eess]. 1

    2024 arXiv

  14. [22]

    LLaV A-med: training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. LLaV A-med: training a large language- and-vision assistant for biomedicine in one day. In Proceed- ings of the 37th International Conference on Neu...

    2023

  15. [23]

    A structure-aware relation network for thoracic diseases detection and segmentation

    Jie Lian, Jingyu Liu, Shu Zhang, Kai Gao, Xiaoqing Liu, Dingwen Zhang, and Yizhou Yu. A structure-aware relation network for thoracic diseases detection and segmentation. IEEE Transactions on Medical Imaging , 40(8):2042–2052,

  16. [24]

    ROUGE: A Package for Automatic Evalu- ation of Summaries

    Chin-Yew Lin. ROUGE: A Package for Automatic Evalu- ation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, 2004. Association for Com- putational Linguistics. 5

    2004

  17. [25]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In Advances in Neural Informa- tion Processing Systems, pages 34892–34916. Curran Asso- ciates, Inc., 2023. 2, 3

    2023

  18. [26]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 26286–26296, Seattle, W A, USA, 2024. IEEE. 3

    2024

  19. [27]

    LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge, 2024. 3

    2024

  20. [28]

    Duy M. H. Nguyen, Hoang Nguyen, Nghiem Diep, Tan Ngoc Pham, Tri Cao, Binh Nguyen, Paul Swoboda, Nhat Ho, Shadi Albarqouni, Pengtao Xie, Daniel Sonntag, and Mathias Niepert. LVM-Med: Learning Large-Scale Self- Supervised Vision Models for Medical Imaging via Second- order Graph...

    2023

  21. [29]

    BLEU: a method for automatic evaluation of ma- chine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of ma- chine translation. In Proceedings of the 40th Annual Meet- ing on Association for Computational Linguistics - ACL ’02, page 311, Philadelphia, Pennsylvania, 2001. Associ...

    2001

  22. [30]

    Castro, Anton Schwaighofer, Matthew P

    Fernando P ´erez-Garc´ıa, Harshita Sharma, Sam Bond- Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Matthew P. Lungren, Maria Teodora Wetscherek, Noel Codella, Stephanie L. Hyland, Javier Alvarez-Valle, and Oz...

    2025

  23. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and others. Learning transferable visual models from natural language supervision. In International conference on machine learn- ...

    2021

  24. [32]

    ReXErr-v1: Clini- cally Meaningful Chest X-Ray Report Errors Derived from MIMIC-CXR

    Vishwanatha Rao, Serena Zhang, Julian Acosta, Subathra Adithan, and Pranav Rajpurkar. ReXErr-v1: Clini- cally Meaningful Chest X-Ray Report Errors Derived from MIMIC-CXR. 2

  25. [33]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  26. [34]

    DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Pa- rameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Pa- rameters. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, pag...

    2020

  27. [35]

    ZeRO-Offload: Democratizing Billion- Scale model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offload: Democratizing Billion- Scale model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) , pages 551–564. USENIX Associatio...

    2021

  28. [36]

    Fink, Moon Kim, Simon Reiß, Ken Herrmann, Jens Kleesiek, and Rainer Stiefelhagen

    Constantin Seibold, Alexander Jaus, Matthias A. Fink, Moon Kim, Simon Reiß, Ken Herrmann, Jens Kleesiek, and Rainer Stiefelhagen. Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via V olumetric Pseudo- Labeling, 2023. arXiv:2306.03934 [eess]. 2, 3

    2023 arXiv

  29. [37]

    Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT

    Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Ng, and Matthew Lungren. Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- c...

    2020

  30. [38]

    CLiP, catheter and line position dataset

    Jennifer S N Tang, Jarrel C Y Seah, Adil Zia, Jay Gajera, Richard N Schlegel, Aaron J N Wong, Dayu Gai, Shu Su, Tony Bose, Marcus L Kok, Alex Jarema, George N Hari- sis, Chris-Tin Cheng, Helen Kavnoudias, Wayland Wang, Anouk Stein, George Shih, Frank Gaillard, Andrew Dixon, an...

    2021

  31. [39]

    Interactive and Explainable Region-guided Radi- ology Report Generation

    Tim Tanida, Philip M ¨uller, Georgios Kaissis, and Daniel Rueckert. Interactive and Explainable Region-guided Radi- ology Report Generation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7433–7442, Vancouver, BC, Canada, 2023. IEEE. 2, 3

    2023

  32. [40]

    Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, Karan Sing- hal, Mike Schaekermann, Rhys May, Roy Lee, SiWai Man, Sara Mahdavi, Zahra Ahmed, Yossi Matias, Joelle Barral...

    2025

  33. [41]

    Sara Mah- davi, Bradley Green, Ewa Dominowska, Blaise Aguera Y Arcas, Joelle Barral, Dale Webster, Greg S

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaeker- mann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, Anil Palepu, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Korn- blith, David Fleet, Philip Mansfield, Sushant Prakash, Renee W...

    2024

  34. [42]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evalua- tion. In 2015 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR) , pages 4566–4575, Boston, MA, USA, 2015. IEEE. 5

    2015

  35. [43]

    Visionllm: Large language model is also an open- ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open- ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36:61501–61513, 2023. 3

    2023

  36. [44]

    The all-seeing project: Towards panop- tic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhen- hang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panop- tic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023. 2, 3

    2023 arXiv

  37. [45]

    A survey of deep- learning-based radiology report generation using multimodal inputs

    Xinyi Wang, Grazziela Figueredo, Ruizhe Li, Wei Emma Zhang, Weitong Chen, and Xin Chen. A survey of deep- learning-based radiology report generation using multimodal inputs. Medical Image Analysis, 103:103627, 2025. 1

    2025

  38. [46]

    Marten: Visual question answering with mask generation for multi-modal document understanding

    Zining Wang, Tongkun Guan, Pei Fu, Chen Duan, Qianyi Jiang, Zhentao Guo, Shan Guo, Junfeng Luo, Wei Shen, and Xiaokang Yang. Marten: Visual question answering with mask generation for multi-modal document understanding. In Proceedings of the Computer Vision and Pattern Recogni...

    2025

  39. [47]

    B. L. Welch. THE GENERALIZATION OF ‘STUDENT’S’ PROBLEM WHEN SEVERAL DIFFERENT POPULA- TION V ARLANCES ARE INVOLVED. Biometrika, 34(1- 2):28–35, 1947. Publisher: Oxford University Press (OUP). 5

    1947

  40. [48]

    Gpt4roi: Instruction tuning large language model on region- of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region- of-interest. In European conference on computer vision , pages 52–70. Springer, 2024. 2, 3

    2024

  41. [49]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gon- zalez, and Ion Stoica. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena, 2023. arXiv:2306.05685 [cs]. 12

    2023 arXiv

  42. [50]

    Topol, and Pranav Rajpurkar

    Hong-Yu Zhou, Juli ´an Nicol ´as Acosta, Subathra Adithan, Suvrankar Datta, Eric J. Topol, and Pranav Rajpurkar. Med- Versa: A Generalist Foundation Model for Medical Image Interpretation, 2025. arXiv:2405.07988 [cs]. 2 Reproducibility Statement For the purpose of reproduction...

    2025 arXiv

  43. [52]

    Can you describe what you see in the image?

  44. [53]

    Please provide an overview of the key observations in the X-ray images

  45. [54]

    What are the significant details captured in this medical image?

  46. [55]

    Summarize the visual findings from this medical scan

  47. [56]

    Give me a brief summary of the image’s diagnostic features

  48. [57]

    Can you outline the main points of interest in this picture?

  49. [58]

    What observations can be made from this radiological image?

  50. [59]

    Describe the significant findings in this visual information

  51. [60]

    Offer a brief overview of the diagnostic details in the picture

  52. [61]

    List the main points of interest in this radiological data

  53. [62]

    Outline the relevant findings of this medical imaging

  54. [63]

    Give me a summarized account of the observations here

  55. [64]

    Provide a concise summary of the diagnostic features

  56. [65]

    Can you identify the key takeaways from this visual data?

  57. [66]

    Highlight the significant findings in this X-ray image

  58. [67]

    Summarize the important aspects of this radiological data

  59. [68]

    Offer a brief synopsis of the observations captured

  60. [69]

    Describe the most salient features in this X-ray image

  61. [70]

    What do you perceive as the primary diagnostic insights from this picture?

  62. [71]

    Provide details about any notable and unremarkable features in the image

  63. [72]

    Describe the overall condition of the subject in the image

  64. [73]

    Can you summarize the key observations from this radiograph?

  65. [74]

    Discuss the significant findings within this X-ray image

  66. [75]

    Brief me on the findings

  67. [76]

    What can you see on the X-ray images?

  68. [77]

    Please provide a summary of the observations made in the images, noting any abnormalities or potential issues

  69. [78]

    Describe what you see in the images and mention if any areas appear normal or unremarkable

  70. [79]

    Summarize the key observations and abnormalities that stand out in the images

  71. [80]

    Give an overview of the findings

  72. [81]

    Summarize the overall impression of the images, emphasizing critical observations

  73. [82]

    Provide a concise summary of the findings using medical jargon

  74. [83]

    Are there any notable or unremarkable findings that should be known to the patient’s primary care physician?

  75. [84]

    Figure 3

    Summarize the findings in a manner that allows for easy communication with the patient’s healthcare team. Figure 3. The user queries uF i ∈ PF for prompting a VLM for radiology findings on an image vi, each paired with the findings rF i as the response to form a single-turn ch...

  76. [2007]

    event- place: Prague, Czech Republic

    Association for Computational Linguistics. event- place: Prague, Czech Republic. 5

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.