Pith. sign in

REVIEW 4 major objections 5 minor 18 references

DermaSynth: Rich Synthetic Image-Text Pairs Using Open Access Dermatology Datasets

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DermaSynth provides 92,020 synthetic image-text pairs, built from 45,205 openly licensed dermatology images, as a foundation for training dermatology vision-language models.

desk verdict A useful new dermatology image-text resource, released openly, but its clinical quality rests on an unverified assumption that Gemini obeys an instruction to ignore the metadata it is given. read the letter →

arxiv 2502.00196 v2 pith:EOXALA6V submitted 2025-01-31 cs.CV cs.AIcs.CL

Abdurrahim Yilmaz , Furkan Yuceyalcin , Ece Gokyayla , Donghee Choi , Ozan Erdem , Ali Anil Demircali , Rahmetullah Varol , Ufuk Gorkem Kirabali
show 3 more authors
Gulsum Gencoglan Joram M. Posma Burak Temelkuran
This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords DermaSynthdermatologyvision-languagemodelssyntheticimage-textpairsinstructiontuningself-instructdermatoscopicimagesopen-accessdatasets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces DermaSynth, a dataset of 92,020 synthetic image-text pairs built from 45,205 openly licensed dermatology images. The authors identify the lack of image-text pairs as the main barrier to vision-language models in dermatology, and their claim is that DermaSynth fills that gap for instruction tuning. Each image is paired with a general clinical question and a metadata-style question, and an advanced multimodal model generates the text answers under an instruction to rely only on visible features. A preliminary fine-tuned model on 5,000 pairs shows a shift toward dermatology-relevant responses. If the dataset holds up under expert review, it gives the field a large public resource for training and evaluating dermatology vision-language models.

What carries the argument

The machinery is the instruction-following generation pipeline. A question-generation stage uses an LLM to produce a diverse question set for each source dataset, blending general clinical prompts with metadata-grounded prompts. Images paired with these questions are then sent to a large multimodal model with a structured prompt that instructs the model to act as a dermatologist, answer solely from visible features, and follow an exact response format. A self-instruct loop is used to expand the seed prompt set iteratively, and post-processing removes nonsensical or hallucinated outputs. This same pipeline is what other researchers would reuse to extend the dataset.

What would settle it

Have dermatologists score a random sample of the 92,020 pairs for whether each answer describes features that are actually visible in the image and are clinically reasonable; a high rate of answers that mention lesions or signs not present in the image would refute the grounding premise. An automated proxy is to generate two versions of the same image-text pair, one with the ground-truth label in the context and one with it withheld, and measure how often the answers coincide or diverge.

Watch

Extended reading notes

Core claim

The paper's central claim is that a large synthetic corpus of 92,020 image-text pairs, curated from 45,205 clinical and dermatoscopic images, can serve as a foundation for fine-tuning dermatology vision-language models. The generation scheme uses broadly diverse prompt types, including observational overviews, diagnosis requests, differential diagnosis, patient explanations, and next steps, together with prompts that incorporate dataset metadata such as label, age, anatomic site, and skin type. The model is prompted with the metadata present in the context but explicitly instructed not to rely on it, so the generated text is meant to describe the visible lesion rather than parrot the label. The paper reports 92,020 pairs after post-processing and a proof-of-concept fine-tuned model on 5,000 pairs from the largest source subset, with example outputs showing more lesion-focused language than the base model. The dataset and model are released for non-commercial research use.

Load-bearing premise

The load-bearing premise is that the large multimodal model's answers are actually based on the image's visible features rather than on the label placed in the prompt, and that those answers are clinically sound despite the absence of expert verification.

Editorial extensions

If this is right

  • Fine-tuning a small vision-language model on 5,000 DermaSynth pairs shifts its outputs toward dermatology-specific, lesion-focused answers compared with the base model.
  • The full 92,020-pair corpus provides a sizeable instruction-tuning resource without requiring new clinical image collection.
  • Because all source images carry permissive open licenses, the synthetic pairs inherit a foundation that is legally usable for research and redistribution.
  • The corpus includes matched pairs that differ only in whether symptom metadata is included, enabling a controlled test of how symptom context changes answers.
  • If the synthetic texts prove visually grounded, the same pipeline can be applied to the other source repositories to enlarge the corpus further.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment the paper does not run is to fine-tune identical models on DermaSynth texts versus label-only captions; the gap in held-out question-answering quality would separate the value of rich synthetic text from the value of the label itself.
  • The non-commercial license, inherited from the upstream model terms, will keep the dataset out of commercial clinical products; replacing the proprietary generator with an open-weight multimodal model would remove that constraint.
  • If expert review confirms clinical accuracy, the same prompt-generation pipeline transfers directly to other image-heavy specialties that also lack image-text pairs, such as ophthalmology or dermatopathology.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces DermaSynth, a dataset of 92,020 synthetic image-text pairs generated from 45,205 dermoscopic and clinical images drawn from five open-access dermatology repositories (DERM12345, BCN20000, PAD-UFES-20, SCIN, and HIBA). Image-text pairs are produced by prompting Gemini 2.0 with general and metadata-based questions using a self-instruct variant, followed by a post-processing step that removes 'unsuitable or nonsensical outputs'. The authors also fine-tune Llama-3.2-11B-Vision-Instruct on 5,000 DERM12345 pairs to produce DermatoLlama 1.0 and release code, prompts, and model weights. The paper reports no quantitative evaluation of the generated text or of the fine-tuned model, and states in Section 6 that expert review of the synthetic content is 'currently underway'.

Significance. If validated, DermaSynth would be a large, permissively licensed resource that directly addresses the scarcity of image-text pairs in dermatology, and the release of code, prompts, and model weights is a practical strength. The scale (92k pairs), the use of open-access sources with clear licensing, and the relatively detailed description of the generation pipeline are the main positive features. However, the paper currently asserts rather than demonstrates clinical quality and visual grounding: the label-leakage risk is central to whether the dataset teaches visual understanding or merely paraphrases metadata, and the reported statistics contain an inconsistency that must be resolved. The absence of any quantitative evaluation, acknowledged by the authors' statement that expert review is still underway, means the resource is promising but not yet ready for use as a validated clinical training set.

major comments (4)
  1. [Section 2 / Appendix 1] The generation prompt places the ground-truth label and patient metadata in the CONTEXT block and instructs Gemini 2.0 to ignore this context, yet no experiment verifies that the generated text is actually grounded in the image. I recommend adding a label-leakage probe (e.g., swapping the label in CONTEXT and measuring how often the answer changes), comparing outputs generated with and without the CONTEXT block, or reporting expert ratings of visual grounding on a random sample. Without such a test, the central claim that DermaSynth provides visually grounded clinical text is unsupported; downstream models trained on this data may learn to copy teacher labels rather than image features. The statement in Section 6 that expert review is 'currently underway' confirms that this validation is not yet available.
  2. [Abstract and Table 1] The abstract states 45,205 images consisting of 13,568 clinical and 35,561 dermatoscopic images, while Table 1 totals 45,205 but lists 12,644 clinical and 32,561 dermatoscopic images; furthermore 13,568 + 35,561 = 49,129, not 45,205. The pair counts also need reconciliation: with 'two questions per image' and 10,000 SCIN images (plus two symptom variants for some samples), the reported 21,859 SCIN pairs require a per-sample breakdown. Please audit all row and column totals and report the exact per-dataset question and pair yields, including how the two SCIN variants are counted.
  3. [Section 5 / Section 2] Section 5 describes DermaSynth as 'quality-checked', but Section 2 defines the post-processing only as removing 'unsuitable or nonsensical outputs such as hallucinated outputs', with no filtering criteria, no counts of discarded samples, no inter-annotator agreement, and no examples of removed outputs. Since the dataset's clinical utility is load-bearing, please operationalize the quality check: specify the filtering rules, report how many candidates were removed and why, and provide at least a sample-based expert assessment (e.g., agreement on a random subset of retained pairs).
  4. [Section 3] The fine-tuned DermatoLlama 1.0 model is showcased with a single illustrative example in Figure 4, but no quantitative results are reported on any benchmark or task. While the dataset is the primary contribution, the model is released as a resource and Figure 4 is presented as evidence of capability; please add at least a small-scale evaluation, such as label accuracy on a held-out portion of DERM12345 or a comparison of the base and fine-tuned models on a fixed set of prompts, so that readers can calibrate the model's behavior.
minor comments (5)
  1. [Abstract] The phrase 'comprising of' should be 'comprising'; additionally, the parenthetical breakdown '13,568 clinical and 35,561 dermatoscopic' is inconsistent with Table 1 and should be corrected.
  2. [Section 4] The model name appears with a typo as 'DermatoLllama 1.0' (three consecutive l's) in the section title and text; this should be corrected for consistency with the HuggingFace repository name.
  3. [Table 1] The use of dashes to denote zero counts is ambiguous; please use explicit '0' entries for the Clinical and Dermatoscopic columns where no images of that type contribute.
  4. [Section 2] There is a missing space in 'wereexcluded' and 'thorough' should be 'through' in the phrase 'refining outputs thorough post-processing stage'.
  5. [Appendix 1] The 'Question Generation Prompt' contains the phrase 'Lets continue enhancing' and 'somethings'; these should be corrected to 'Let's' and 'something' for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: DermaSynth is generated from external image repositories and an external LLM; the label-in-context prompt creates a validity/leakage risk, not a by-construction equivalence.

full rationale

The paper's central output is a dataset, not a derived quantitative prediction. DermaSynth is produced by sending images, metadata, and hand-selected prompts to Gemini 2.0, an external model, and then post-processing the outputs; no fitted parameter is renamed as a prediction and no numerical benchmark is claimed. DermatoLlama 1.0 is explicitly preliminary and only illustrated with an example (Figure 4), so there is no measured performance claim that could reduce to the training data by construction. The self-citations are [10], a BioRender figure credit, and [11], the DERM12345 source dataset; these are source/attribution citations, not load-bearing uniqueness theorems or unverified premises invoked to forbid alternatives. The most plausible circularity concern is that the API prompt places the ground-truth label and metadata in the CONTEXT block and instructs Gemini to ignore it; however, the paper does not define any claimed result as the output of that prompt, and it explicitly discloses in Section 6 that expert review is 'currently underway.' That is a stated limitation and a label-leakage/validation risk, not a demonstrated equivalence between input and output. Therefore no circular step meeting the quoted-evidence standard is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The ledger captures the hand-chosen generation settings and the unverified assumptions about the teacher model and filtering. No new physical or conceptual entities are introduced.

free parameters (3)
  • questions_per_image = 2
    Each image receives exactly two prompts (one general, one metadata-based); this is a hand-chosen design decision, not derived from data.
  • fine_tune_sample_size = 5,000
    The proof-of-concept model is trained on a manually selected 5,000 pairs from DERM12345; chosen without a scaling study.
  • prompt_variations_per_type = 20
    The question pool includes 120 question types and 20 variations per type; these counts are designer choices.
assumptions (3)
  • domain assumption Gemini 2.0 can produce visually grounded, clinically accurate dermatology descriptions from images when given the provided prompts.
    The entire dataset quality rests on the teacher model's ability; the paper provides no external validation.
  • ad hoc to paper Including metadata in the prompt while instructing the model to ignore it prevents label leakage.
    Appendix 1 instructs the model not to rely on metadata, but there is no test that the answers are not influenced by the label.
  • ad hoc to paper The post-processing step removes hallucinations and nonsensical outputs.
    Section 2 mentions removal of unsuitable outputs but does not define criteria, so this is an asserted filtering step with no verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DermaSynth: Rich Synthetic Image-Text Pairs Using Open Access Dermatology Datasets." pith.science (2026). https://pith.science/paper/EOXALA6V

@misc{pith2026250200196,
  author       = {Pith},
  title        = {Pith review of: DermaSynth: Rich Synthetic Image-Text Pairs Using Open Access Dermatology Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOXALA6V}},
  note         = {Machine review of arXiv:2502.00196}
}
read the original abstract

A major barrier to developing vision large language models (LLMs) in dermatology is the lack of large image--text pairs dataset. We introduce DermaSynth, a dataset comprising of 92,020 synthetic image--text pairs curated from 45,205 images (13,568 clinical and 35,561 dermatoscopic) for dermatology-related clinical tasks. Leveraging state-of-the-art LLMs, using Gemini 2.0, we used clinically related prompts and self-instruct method to generate diverse and rich synthetic texts. Metadata of the datasets were incorporated into the input prompts by targeting to reduce potential hallucinations. The resulting dataset builds upon open access dermatological image repositories (DERM12345, BCN20000, PAD-UFES-20, SCIN, and HIBA) that have permissive CC-BY-4.0 licenses. We also fine-tuned a preliminary Llama-3.2-11B-Vision-Instruct model, DermatoLlama 1.0, on 5,000 samples. We anticipate this dataset to support and accelerate AI research in dermatology. Data and code underlying this work are accessible at https://github.com/abdurrahimyilmaz/DermaSynth.

Figures

Figures reproduced from arXiv: 2502.00196 by the authors.

Figure 1
Figure 1. Overview of the synthetic data creation process for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The most common 20 root verb-noun pairs of question. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A figure from DERM12345 dataset with a field specific question and a dataset [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Shows input images with a simple prompt and their answers by original Llama [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages

  1. [1]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 , 2023

    arXiv 2023

  2. [2]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024. 8

    work page 2024

  3. [3]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296–26306, 2024

    2024

  4. [4]

    Pmc-llama: toward building open-source language models for medicine

    Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmc-llama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, page ocae045, 2024

    work page 2024

  5. [5]

    Medalpaca–an open- source collection of medical conversational ai models and training data

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Ober- hauser, Alexander L¨ oser, Daniel Truhn, and Keno K Bressem. Medalpaca–an open- source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023

    arXiv 2023

  6. [6]

    Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities, 2023

    Chang Shu, Baian Chen, Fangyu Liu, Zihao Fu, Ehsan Shareghi, and Nigel Collier. Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities, 2023

    work page 2023

  7. [7]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Infor- mation Processing Systems, 36, 2024

    work page 2024

  8. [8]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction- following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    work page 2023

Show all 18 references
  1. [9]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022

    2022

  2. [10]

    Created in biorender

    Abdurrahim Yilmaz and Burak Temelkuran. Created in biorender. https:// BioRender.com/f99q320, 2025

    2025

  3. [11]

    Derm12345: A large, multisource dermatoscopic skin lesion dataset with 40 subclasses

    Abdurrahim Yilmaz, Sirin Pekcan Yasar, Gulsum Gencoglan, and Burak Temelkuran. Derm12345: A large, multisource dermatoscopic skin lesion dataset with 40 subclasses. Scientific Data , 11(1):1302, 2024

    2024

  4. [12]

    Bcn20000: Dermoscopic lesions in the wild

    Carlos Hern´ andez-P´ erez, Marc Combalia, Sebastian Podlipnik, Noel CF Codella, Veron- ica Rotemberg, Allan C Halpern, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Brian Helba, et al. Bcn20000: Dermoscopic lesions in the wild. Scientific Data , 11(1):641, 2024

    2024

  5. [13]

    Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones

    Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G de Angelo, F´ abio CR Alves Jr, Jos´ e GM Esgario, Alana C Simora, Pedro BC Castro, et al. Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from...

    2020

  6. [14]

    9 Crowdsourcing dermatology images with google search ads: Creating a real-world skin condition dataset

    Abbi Ward, Jimmy Li, Julie Wang, Sriram Lakshminarasimhan, Ashley Carrick, Bilson Campana, Jay Hartford, Tiya Tiyasirichokchai, Sunny Virmani, Renee Wong, et al. 9 Crowdsourcing dermatology images with google search ads: Creating a real-world skin condition dataset. arXiv prep...

    2024 arXiv

  7. [15]

    A dataset of skin lesion images collected in argentina for the evaluation of ai tools in this population

    Mar ´ ıa Agustina Ricci Lara, Mar ´ ıa Victoria Rodr ´ ıguez Kowalczuk, Maite Lisa Eliceche, Mar ´ ıa Guillermina Ferraresso, Daniel Roberto Luna, Sonia Elizabeth Benitez, and Luis Daniel Mazzuoccolo. A dataset of skin lesion images collected in argentina for the evaluation of...

    2023

  8. [16]

    Exploring the role of large language models in melanoma: A systematic review

    Mor Zarfati, Girish N Nadkarni, Benjamin S Glicksberg, Moti Harats, Shoshana Green- berger, Eyal Klang, and Shelly Soffer. Exploring the role of large language models in melanoma: A systematic review. Journal of Clinical Medicine , 13(23):7480, 2024

    2024

  9. [17]

    Mpoxvlm: A vision- language model for diagnosing skin lesions from mpox virus infection

    Xu Cao, Wenqian Ye, Kenny Moise, and Megan Coffee. Mpoxvlm: A vision- language model for diagnosing skin lesions from mpox virus infection. arXiv preprint arXiv:2411.10888, 2024

    2024 arXiv

  10. [18]

    What is this image

    Ryutaro Tanno, David GT Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, et al. Collaboration between clinicians and vision–language models in radiology report gener- ation. Nature Medicine, pages...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.