Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination

Kayhan Batmanghelich; Li Sun; Xinliu Zhong

REVIEW 5 major objections 4 minor 21 references

Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding perturbed-report discrimination to contrastive pretraining makes biomedical vision-language representations sensitive to clinical sentence structure, and reports gains on chest X-ray classification, NLI, and a zero-shot ranking task.

desk verdict Plausible pre-training recipe; the central semantic-structure evaluation is circular, so the headline claims are not yet supported. read the letter →

arxiv 2506.01902 v1 pith:MRRZTGYS submitted 2025-06-02 cs.CV cs.CL

Xinliu Zhong , Kayhan Batmanghelich , Li Sun This is my paper

classification cs.CVcs.CL

keywords biomedicalvision-languagepretrainingcontrastivelearningradiologyreportperturbationlocalattentivelossclinicalsentencestructureself-supervisedchestX-rayimagingmedicalnaturallanguageinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that standard global contrastive pretraining misses how clinical text conveys meaning through word order and structure. Its proposed remedy adds a report perturbation discrimination loss that forces the model to rank the original radiology report above nine perturbed versions of the same words, alongside a local attentive contrastive loss that aligns image sub-regions with word-level text features. If the paper is right, biomedical vision-language models can learn compositional clinical semantics without manual annotations, improving label-efficient image classification and clinical text understanding. The authors report gains over two strong baselines in chest X-ray classification, radiology-specific natural language inference, and their zero-shot structure task, with mixed results on a general clinical NLI benchmark.

What carries the argument

The central object is the perturbation set: nine hand-designed text transformations that preserve the exact words of a radiology report while degrading its sentence structure (e.g., shuffling all words, reversing the sentence, shuffling trigrams, shuffling nouns and adjectives, or replacing adjectives with antonyms). Around this, the model combines three contrastive losses: a global image-report matching loss, a local attentive contrastive loss that computes attention-weighted sums of image sub-region features for each word and aggregates the matching scores, and a perturbation discrimination loss that scores the original report against the nine perturbed texts. The final objective is $L = L_{\mathrm{global}} + \alpha L_{\mathrm{local}} + \beta L_{\mathrm{pert}}$ with $\alpha=\beta=0.1$ and temperature $\tau=0.07$, which forces the text encoder to attend to ordering and structure as well as lexical content.

What would settle it

Run the trained model on perturbed reports generated by rules that were held out during training, for example synonym substitution, inserted negations, or shuffles at a different granularity. If zero-shot accuracy on those unseen perturbation types falls to chance while accuracy on the original nine stays high, the model has learned the nine transformations themselves rather than general clinical sentence-structure sensitivity.

Watch

Extended reading notes

Core claim

The paper claims that global contrastive alignment alone treats radiology text like a bag of words, missing the clinical meaning carried by word order and sentence structure. Its proposed model adds a report perturbation sensitivity loss: for each image-report pair, nine texts are generated from the report by shuffling words, swapping adjacent tokens, reversing the sentence, shuffling within and across trigrams, shuffling parts of speech, or replacing adjectives with antonyms, while keeping the same words; the model must give the original report the highest similarity to the image. On top of this, a local attentive contrastive loss aligns attention-weighted image sub-regions with word-level text embeddings. With both losses added to the global contrastive loss, the paper reports average accuracy improvements on multi-task chest X-ray classification (72.6% versus 50.0% and 36.2% for the two baselines), a small gain on radiology-specific NLI, and 49.0% accuracy on its zero-shot semantic-structure task against 44.3% and 43.1% for the baselines; on general clinical NLI the full model trails the strongest baseline, so the claimed advantage is concentrated on tasks close to the perturbation signal.

Load-bearing premise

The load-bearing premise is that the nine hand-designed perturbations measure general clinical sentence-structure understanding, but because the same perturbations are used in training and in the zero-shot evaluation, the reported 49.0% accuracy may only show that the model learned to recognize those exact negative transformations.

Editorial extensions

If this is right

A model trained this way should rank a correctly structured report above word-identical shuffled versions even when the bag of words is the same.
The local attentive loss should improve label-efficient recognition of findings that occupy small image sub-regions, such as consolidation and pleural effusion.
The perturbation protocol is a fully self-supervised training signal, so it can be applied to other paired image-report corpora without annotations.
The same nine perturbation types can also serve as a reusable zero-shot evaluation for compositional understanding of clinical text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We would expect the 49.0% zero-shot accuracy to overstate generalization, because the evaluation reuses the same perturbation rules used in training; a fair test needs unseen perturbation types.
Extending the perturbation set beyond rewrites of the same words, for example by substituting synonyms or inserting negations, would test whether the model learns semantic content rather than surface order.
The perturbed-report discrimination idea could transfer to other clinical settings where text is paired with structured data, such as discharge summaries paired with diagnostic codes, as a cheap structure-sensitivity signal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. The method is a sensible combination of established ingredients: GLoRIA's local attentive loss and NegCLIP's perturbed-caption discrimination, with nine hand-designed radiology perturbations. The main evidence for the central claim is circular: the Open-I semantic structure evaluation uses the same nine perturbations as the training loss, so the reported 49% accuracy mostly measures how well the model fit the pretext task.

What the paper does well: it is clearly written, the perturbation set is concrete and useful, and the comparison protocol is fair—same pretraining data, same unimodal encoders, same setup as ConVIRT and GLoRIA. Reporting a version without the local loss is also a good instinct, even though that ablation is incomplete.

The soft spots are serious. The circularity is load-bearing, and the paper itself states in Section III-A-3-a that the perturbations are used both in the loss and in subsequent evaluation. That kills the zero-shot semantic structure result as evidence of generalization. The w/o local loss row still includes the perturbation loss, so there is no beta=0 ablation isolating the new component. CheXpert is non-standard: only two of fourteen labels, 500 test images, no error bars, and the numbers swing so much that adding the local loss moves consolidation from 33.8% to 93.4% while dropping pleural effusion from 77.8% to 51.8%. And the full model trails ConVIRT on MedNLI. The abstract's claim about 'more semantic meaningful representations' is not supported by these data.

Bottom line: the idea is worth referee time. The fix is straightforward—evaluate on unseen perturbations or a genuinely independent benchmark, run a beta=0 ablation, and use a more standard CheXpert protocol. I would send this to review rather than desk reject, but acceptance would need major revision. For my own work, I would not cite it yet; I would keep it in mind as a related approach if the independent evaluation pans out.

Referee Report

5 major / 4 minor

Summary. The paper proposes a self-supervised biomedical vision-language pretraining method that combines a global image-text contrastive loss, a local attentive contrastive loss, and a newly introduced report perturbation sensitivity loss. The model is pretrained on Open-I chest X-ray image-report pairs and evaluated on MedNLI/RadNLI text classification, CheXpert multi-task image classification, and a custom 'clinical semantic structure evaluation' on Open-I. The authors claim that their approach learns more semantically meaningful and robust multi-modal representations and outperforms ConVIRT and GLoRIA.

Significance. The underlying idea of using hand-designed text perturbations to make a vision-language model sensitive to sentence structure is relevant and potentially useful for biomedical domains where negation and word order carry clinical meaning. However, the paper's central claim is not supported by the reported experiments. The main evaluation is circular, the CheXpert protocol is non-standard and results are unstable, and the ablation does not isolate the contribution of the perturbation loss. If the evaluation were independent and the ablations complete, the method could be of interest, but in its current form the evidence is not convincing.

major comments (5)

[III-A-3-a and Table IV] The zero-shot clinical semantic structure evaluation is circular. Section III-A-3-a states that the nine perturbations 'are used both in the calculation of report perturbation sensitivity loss and subsequent evaluation.' The model is trained to prefer the original report over these exact nine perturbation types via Lpert, and then Table IV reports accuracy on the same discrimination task. The 49.0% accuracy therefore measures how well the model fits the training objective, not whether it has learned generalizable compositional understanding. To support the central claim, the evaluation must use held-out perturbation types or an independent compositional benchmark (e.g., previously unseen corruptions or a task not tied to the training loss).
[IV-C-2 and Table II] The CheXpert evaluation is non-standard and does not substantiate the claim of strong cross-modal correlation. Only 2 of the 14 CheXpert labels are used, the metric is accuracy on a 500-image test set with no error bars or statistical significance, and the per-task results are unstable: adding the local loss improves consolidation from 33.8% to 93.4% while decreasing pleural effusion from 77.8% to 51.8%. This pattern suggests high variance or overfitting to the small test set. The authors should follow the standard CheXpert benchmark protocol (all 14 labels, AUC, proper patient-wise split) or at minimum report confidence intervals and significance tests.
[IV-C-1 and Table III] The text classification results contradict the claim of improved clinical language understanding. The full model (Ours) underperforms ConVIRT on MedNLI (85.79% vs 86.80%), and the variant without the local loss achieves the highest MedNLI accuracy (87.62%). The RadNLI improvement over ConVIRT is marginal (68.96% vs 68.50%). Moreover, the 'w/o local loss' row still includes the perturbation loss, so it does not isolate the effect of Lpert. A proper ablation should include a beta=0 run (global only) and a run with global+local (no Lpert) to attribute any improvement to the proposed perturbation loss.
[III-A-3-a and Table I] The description of perturbations is internally inconsistent. The text says the perturbations 'keep the same words' but the 'Replace Adjectives with Antonyms' perturbation replaces 'clear' with 'unclear', changing the words. This discrepancy affects the interpretation of the perturbation set as structure-preserving versus word-changing. Additionally, the choice of these specific nine perturbations is not justified; it is unclear why this set is comprehensive or representative enough to evaluate 'semantic structure' understanding.
[IV-A-1 and IV-C] The number of pretraining pairs is inconsistent: Section IV-A-1 reports 6,469 image-text pairs after removing empty values, while Section IV-C states 'using the same 6000 Image-Text pairs of the Open-I dataset as input.' This discrepancy needs clarification, as it affects reproducibility. Also, the hyperparameters alpha, beta, and tau are said to be chosen by grid search, but no search range or selection criterion is given.

minor comments (4)

[Abstract and I] The phrase 'manipulates clinical texture well' (Section I) is unclear; it probably means 'clinical text' or 'clinical semantics'. Please rephrase.
[III-B-2] The local attentive contrastive loss equations contain typographical issues (e.g., "WX" and mismatched indices). Please rewrite them clearly.
[IV-C-3] The 'clinical semantic structure evaluation' as described is essentially image-text retrieval with hard negatives; it is not a new task. The name overstates its novelty.
[References] Some references are incomplete or lack venue information (e.g., [5] and [16]). Please ensure all references are properly formatted.

Circularity Check

1 steps flagged · score 6.0 of 10

Open-I semantic-structure evaluation reuses the exact nine perturbations optimized by the Lpert loss, so the reported 49% chiefly measures pretext-task fit rather than generalizable compositional understanding.

fitted input called prediction [Section III-A-3-a (Generating Text Perturbations), Section III-A-3-b (Contrastive Loss), Section IV-C-3 (Clinical Semantic Structure Evaluation), Tables I and IV]
"These perturbations, shown in Table I, are used both in the calculation of report perturbation sensitivity loss and subsequent evaluation. ... In our clinical semantic structure evaluation task, we used the aforementioned text perturbations as negatives. ... If the similarity between the original aligned Image-Text pair is the highest, then the decision is considered as correct."

The report perturbation sensitivity loss Lpert trains the model to choose the original report over exactly the nine generated perturbation types for a given image. The Open-I semantic structure evaluation then computes whether the original report has the highest cosine similarity against exactly the same nine perturbation types. Hence Table IV's accuracy is essentially a hard-accuracy version of the Lpert training objective applied to a held-out split; it measures how well the model fits the pretext task, not whether the representation generalizes to unseen semantic corruptions. The row 'Ours (w/o local loss)' still includes Lpert, so no beta=0 ablation isolates the perturbation loss, and the 'zero-shot' framing is misleading because the perturbation types are not held out.

full rationale

The paper's central circularity is confined to the clinical semantic structure evaluation: the same nine perturbations used to generate negatives for Lpert are reused verbatim as the negatives in the Open-I evaluation. Because Lpert is a cross-entropy loss that prefers the original report over these nine variants, and the evaluation is an accuracy measure of exactly that preference, Table IV cannot support the claim of generalizable semantic understanding; it primarily tracks optimization of the training loss on the pretext distribution. The external benchmarks provide some independent content, but they are mixed (the full model underperforms ConVIRT on MedNLI, 85.79% vs 86.80%) and the CheXpert protocol is non-standard with only two tasks and a 500-image test set. No load-bearing self-citation chain or imported uniqueness theorem is present. The semantic-structure claim therefore partially reduces by construction, while the paper retains some independent downstream evaluation, so a score of 6 is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the perturbation negatives, the quality of the pretrained CXR-BERT encoder, and the correctness of the image-text pairing. The main non-standard ingredient is the hand-curated set of 9 perturbation templates, which is a free design choice and also the basis of the circular evaluation.

free parameters (4)

alpha (local loss weight) = 0.1
Scaling factor for L_local in the final loss; set by targeted grid search (Section IV-B).
beta (perturbation loss weight) = 0.1
Scaling factor for L_pert; set by grid search (Section IV-B).
temperature tau = 0.07
Temperature for all contrastive losses; set by grid search and typical CLIP value.
perturbation template set = 9 hand-curated transformations
The specific set of shuffling and antonym replacements determines what negatives the model sees and what the evaluation measures; it is chosen by the authors, not derived from theory.

assumptions (4)

domain assumption CXR-BERT, pretrained on PubMed, MIMIC-III, and MIMIC-CXR, provides domain-specific text embeddings that already capture radiology semantics.
The method loads a published CXR-BERT checkpoint (Section III-A) and assumes its representations are a faithful text encoder; if this pretraining is inadequate, the fine-grained contrastive losses are built on a weak base.
ad hoc to paper The nine perturbations in Table I preserve the same words (except antonym replacement) and serve as valid negatives for the associated image; a perturbed report is always less faithful to the image than the original.
The loss and the Open-I evaluation treat every perturbation as a negative. Some perturbations, e.g., trigram shuffles, may accidentally retain a plausible reading, adding label noise; antonym replacement changes tokens, conflicting with the same-words claim in the abstract.
domain assumption The Open-I training set pairs each image with the correct findings section; no filtering for ambiguous or contradictory reports is described.
Correct pairing is the supervision for all three losses; mislabeled pairs would teach the model to prefer wrong text.
standard math Standard properties of the InfoNCE/cross-entropy contrastive losses are assumed, e.g., that the gradient drives matched pairs together and unmatched pairs apart.
The losses are standard softmax cross-entropy over batch negatives; no proof of convergence or optimality is needed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination." pith.science (2026). https://pith.science/paper/MRRZTGYS

@misc{pith2026250601902,
  author       = {Pith},
  title        = {Pith review of: Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MRRZTGYS}},
  note         = {Machine review of arXiv:2506.01902}
}

read the original abstract

Vision-language models pre-trained on large scale of unlabeled biomedical images and associated reports learn generalizable semantic representations. These multi-modal representations can benefit various downstream tasks in the biomedical domain. Contrastive learning is widely used to pre-train vision-language models for general natural images and associated captions. Despite its popularity, we found biomedical texts have complex and domain-specific semantics that are often neglected by common contrastive methods. To address this issue, we propose a novel method, perturbed report discrimination, for pre-train biomedical vision-language models. First, we curate a set of text perturbation methods that keep the same words, but disrupt the semantic structure of the sentence. Next, we apply different types of perturbation to reports, and use the model to distinguish the original report from the perturbed ones given the associated image. Parallel to this, we enhance the sensitivity of our method to higher level of granularity for both modalities by contrasting attention-weighted image sub-regions and sub-words in the image-text pairs. We conduct extensive experiments on multiple downstream tasks, and our method outperforms strong baseline methods. The results demonstrate that our approach learns more semantic meaningful and robust multi-modal representations.

Figures

Figures reproduced from arXiv: 2506.01902 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

[1]

Deep learning in medical image analysis: A third eye for doctors,

A. Fourcade and R. H. Khonsari, “Deep learning in medical image analysis: A third eye for doctors,” Jour- nal of stomatology, oral and maxillofacial surgery , vol. 120, no. 4, pp. 279–288, 2019

work page 2019
[2]

Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,

Y . Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 9, pp. 2251–2265, 2018

2018
[3]

Learning transferable visual models from natural language super- vision,

A. Radford, J. W. Kim, C. Hallacy, et al. , “Learning transferable visual models from natural language super- vision,” International conference on machine learning , pp. 8748–8763, 2021

work page 2021
[4]

Scaling up visual and vision-language representation learning with noisy text supervision,

C. Jia, Y . Yang, Y . Xia, et al. , “Scaling up visual and vision-language representation learning with noisy text supervision,” International conference on machine learning, pp. 4904–4916, 2021

work page 2021
[5]

Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing,

B. Boecking, N. Usuyama, S. Bannur, et al. , “Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing,” en, arXiv.org, Apr. 2022. DOI: 10.1007/978-3-031-20059-5 1

doi:10.1007/978-3-031-20059-5 2022
[6]

When and why vision-language models behave like bags-of-words, and what to do about it?

M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” The Eleventh International Conference on Learning Representations, 2022

work page 2022
[7]

Lessons from natural lan- guage inference in the clinical domain,

A. Romanov and C. Shivade, “Lessons from natural lan- guage inference in the clinical domain,” arXiv preprint arXiv:1808.06752, 2018

arXiv 2018
[8]

Improving factual completeness and con- sistency of image-to-text radiology report generation,

Y . Miura, Y . Zhang, E. B. Tsai, C. P. Langlotz, and D. Jurafsky, “Improving factual completeness and con- sistency of image-to-text radiology report generation,”

work page

Show all 21 references

[9]

Chexpert: A large chest radiograph dataset with uncertainty labels and ex- pert comparison,

J. Irvin, P. Rajpurkar, M. Ko, et al., “Chexpert: A large chest radiograph dataset with uncertainty labels and ex- pert comparison,” Proceedings of the AAAI conference on artificial intelligence , vol. 33, no. 01, pp. 590–597, 2019

2019
[10]

Contrastive learning of medical visual representations from paired images and text,

Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” Machine Learning for Healthcare Conference , pp. 2–25, 2022

2022
[11]

Joint learning of localized representations from medical im- ages and reports,

P. M ¨uller, G. Kaissis, C. Zou, and D. Rueckert, “Joint learning of localized representations from medical im- ages and reports,” in 2022, pp. 685–701

2022
[12]

Medclip: Contrastive learning from unpaired medical images and text,

Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,” arXiv preprint arXiv:2210.10163 , 2022

2022 arXiv
[13]

Expert-level detection of patholo- gies from unannotated chest x-ray images via self- supervised learning,

E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y . Ng, and P. Rajpurkar, “Expert-level detection of patholo- gies from unannotated chest x-ray images via self- supervised learning,” Nature Biomedical Engineering , vol. 6, no. 12, pp. 1399–1406, 2022

2022
[14]

Pubmed data download,

“Pubmed data download,” en, PubMed, [Online]. Avail- able: https://pubmed.ncbi.nlm.nih.gov/download/
[15]

MIMIC-III, a freely accessible critical care database,

A. E. W. Johnson, T. J. Pollard, L. Shen, et al. , “MIMIC-III, a freely accessible critical care database,” en, Scientific Data, vol. 3, no. 1, p. 160 035, May 2016, Number: 1 Publisher: Nature Publishing Group, ISSN : 2052-4463. DOI: 10.1038/sdata.2016.35

2016 doi
[16]

Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition,

S.-C. Huang, L. Shen, M. P. Lungren, and S. Ye- ung, “Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition,” 2021, pp. 3942–3951

2021
[17]

Spacy 2: Natural lan- guage understanding with bloom embeddings, convolu- tional neural networks and incremental parsing. neural machine translation,

M. Honnibal and I. Montani, “Spacy 2: Natural lan- guage understanding with bloom embeddings, convolu- tional neural networks and incremental parsing. neural machine translation,” 2017, pp. 688–697

2017
[18]

What context features can transformer language models use?

J. O’Connor and J. Andreas, “What context features can transformer language models use?” Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)...

2021
[19]

Design and development of a multimodal biomedical information retrieval system,

D. Demner-Fushman, S. Antani, M. Simpson, and G. Thoma, “Design and development of a multimodal biomedical information retrieval system,” Journal of Computing Science and Engineering , vol. 6, Jun. 2012. DOI: 10.5626/JCSE.2012.6.2.168

2012 doi
[20]

An overview of gradient descent optimiza- tion algorithms,

S. Ruder, “An overview of gradient descent optimiza- tion algorithms,” 2017. arXiv: 1609.04747 [cs.LG]

2017 arXiv
[2021]

arXiv: 2010.10042 [cs.CL]

2010 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Deep learning in medical image analysis: A third eye for doctors,

A. Fourcade and R. H. Khonsari, “Deep learning in medical image analysis: A third eye for doctors,” Jour- nal of stomatology, oral and maxillofacial surgery , vol. 120, no. 4, pp. 279–288, 2019

work page 2019

[2] [2]

Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,

Y . Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 9, pp. 2251–2265, 2018

2018

[3] [3]

Learning transferable visual models from natural language super- vision,

A. Radford, J. W. Kim, C. Hallacy, et al. , “Learning transferable visual models from natural language super- vision,” International conference on machine learning , pp. 8748–8763, 2021

work page 2021

[4] [4]

Scaling up visual and vision-language representation learning with noisy text supervision,

C. Jia, Y . Yang, Y . Xia, et al. , “Scaling up visual and vision-language representation learning with noisy text supervision,” International conference on machine learning, pp. 4904–4916, 2021

work page 2021

[5] [5]

Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing,

B. Boecking, N. Usuyama, S. Bannur, et al. , “Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing,” en, arXiv.org, Apr. 2022. DOI: 10.1007/978-3-031-20059-5 1

doi:10.1007/978-3-031-20059-5 2022

[6] [6]

When and why vision-language models behave like bags-of-words, and what to do about it?

M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” The Eleventh International Conference on Learning Representations, 2022

work page 2022

[7] [7]

Lessons from natural lan- guage inference in the clinical domain,

A. Romanov and C. Shivade, “Lessons from natural lan- guage inference in the clinical domain,” arXiv preprint arXiv:1808.06752, 2018

arXiv 2018

[8] [8]

Improving factual completeness and con- sistency of image-to-text radiology report generation,

Y . Miura, Y . Zhang, E. B. Tsai, C. P. Langlotz, and D. Jurafsky, “Improving factual completeness and con- sistency of image-to-text radiology report generation,”

work page