DermAI: Clinical dermatology acquisition through quality-driven image collection for AI classification in mobile

Emanoel Thyago; \'Erico Medeiros; F\'abio Papais; Francisco Mauro; J\'essica Guido; Kelvin Cunha; Nat\'alia Lopes; Paulo Borba; Rodrigo Abreu; Shirley Cruz

REVIEW 4 major objections 3 minor 8 references

DermAI: Clinical dermatology acquisition through quality-driven image collection for AI classification in mobile

T0 review · 4 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Quality-controlled smartphone photos make skin-lesion AI transfer across clinical datasets better than public training data alone.

desk verdict Useful acquisition tool and promising dataset, but the main generalization claim currently rests on a possibly leaky split and inconsistent dataset counts. read the letter →

arxiv 2511.10367 v2 pith:JODX3CIO submitted 2025-11-13 cs.CV cs.AI

Thales Bezerra , Emanoel Thyago , Kelvin Cunha , Rodrigo Abreu , Fábio Papais , Francisco Mauro , Natália Lopes , Érico Medeiros

show 4 more authors

Jéssica Guido Shirley Cruz Paulo Borba Tsang Ing Ren

This is my paper · ORCID

classification cs.CVcs.AI

keywords DermAIskinlesionclassificationmobilehealthdatasetacquisitionimagequalityassessmentfine-tuningcross-datasetgeneralizationclinicaldermatology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DermAI, a smartphone app that standardizes how skin lesions are photographed during routine consultations and checks image quality on the device before storage. Using this app, the authors built a dataset of 3,401 images covering 200 lesions from a Brazilian public hospital, spanning diverse skin tones and devices. They report that models trained on a public dataset perform poorly when tested on their new data, while fine-tuning the same architectures on DermAI's quality-controlled images substantially improves accuracy and recall on several clinical datasets. The central claim is that controlled acquisition plus local fine-tuning matters more than simply adding more public data, a finding that would shape how clinical AI is trained in low-resource settings.

What carries the argument

The DermAI acquisition pipeline: a smartphone app that enforces standardized capture (on-screen centering guides, ~5 cm distance, square crop preview) and runs a lightweight MobileNetV3-based image-quality model that outputs four binary indicators (sharpness, blur, exposure, compression) to reject degraded photos in real time. The mechanism works by preventing low-quality images from entering the dataset, then using that clean dataset to fine-tune CNN backbones locally; ensembles of these backbones are combined by majority vote or a learned fusion MLP.

What would settle it

Request the split metadata or lesion identifiers for the DermAI dataset and check whether any lesion appears in both the training and test partitions; if overlap exists, recompute the DermAI test scores using a lesion-aware split and observe whether accuracy drops substantially.

Watch

Extended reading notes

Core claim

On the paper's own terms, the core discovery is that models trained on large public smartphone datasets fail to generalize to a clinically collected, quality-controlled dataset, whereas retraining the same lightweight CNNs on DermAI's data improves cross-dataset performance. The best results come from combining a quality-filtered version of the public dataset with DermAI's images, suggesting that curated, context-specific acquisition and quality filtering are decisive for transfer. The paper also shows that an ensemble using learned fusion of multiple backbones yields the highest scores, especially for malignant-suspect cases, and that adding a noisy public dataset to training does not help.

Load-bearing premise

The dataset contains multiple photos per lesion, so the 80/10/10 split must be made on unique lesions rather than individual images; otherwise the same lesion could appear in both training and test sets, inflating the reported accuracy.

Editorial extensions

If this is right

If replicated, this workflow gives clinics a practical method to build small, high-quality local datasets that outperform larger public datasets for their own populations.
Quality filtering of existing public datasets (the paper's 'PAD filter' experiment) can yield accuracy gains even with fewer training samples, suggesting curation should be a standard preprocessing step.
Lightweight models fine-tuned on local data can run on-device, enabling real-time triage during consultations and continuous collection of feedback-labelled images.
The learned fusion ensemble demonstrates that combining multiple architectures on small local datasets can improve sensitivity for malignant cases, a key safety metric.
The finding that adding a third noisy dataset did not help implies that dataset quality and relevance, not sheer size, determine clinical transferability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not state whether its 80/10/10 split separates unique lesions or just images; if images from the same lesion appear in both training and test partitions, the within-DermAI accuracy could be inflated by memorization rather than generalization.
A natural extension would be a per-lesion leave-one-out evaluation to establish whether the reported cross-dataset gains persist when the same lesion is not shared between train and test.
The on-device quality model was trained on synthetic distortions; real-world artifacts such as ink markings, hair, or reflections from diverse cameras may require calibration against clinical rater judgment before deployment.
The findings suggest a general recipe for under-served settings: collect a modest number of quality-controlled local images with clinician review, fine-tune a small backbone, and use ensemble fusion, rather than relying on public repositories built elsewhere.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The paper you'll want to know about: DermAI is a smartphone app for standardized skin-lesion image acquisition with on-device quality checks, plus a new Brazilian clinical dataset. The genuinely new part is the dataset and the experiment showing that quality-filtering an existing public dataset (PAD-UFES-20) and retraining improves cross-dataset performance. That is a practical, useful result for deployment in diverse populations, and the authors are right that acquisition standardization matters.

What it does well: the app workflow is sensible—capture, quality validation, annotation, and local model adaptation. The cross-dataset evaluation protocol (train on PAD, test on DDI and DermAI; train on DermAI, test on the others) is a good way to expose generalization problems. The decision to focus on lightweight models is appropriate for mobile deployment.

The soft spots are real, though. The central generalization claim rests on the DermAI test numbers in Table 1, but the paper never says whether the 80/10/10 split is done at the image or lesion level. With 3,401 images of 200 lesions, about 17 per lesion, an image-level split will leak: photos of the same lesion end up in both training and test, and the model can memorize lesion-specific features. If that is the case, the 0.80 accuracy on DermAI is inflated, and the comparison against PAD-trained models becomes unfair. This is not a minor detail; it is the load-bearing piece of the abstract's claim. The authors need to state the split is lesion-exclusive and ideally report metrics at the lesion level.

There is also an internal inconsistency in the dataset statistics. They report 3,401 images with 2,273 benign, 608 malignant, 520 pre-malignant, which sums to 3,401. But the subclass counts (melanoma 30 + BCC 477 + SCC 142 + nevus 568 + AK 520 + benign keratosis 706 + solar lentigo 285) sum to 2,728. That is 673 images unaccounted for. This makes the dataset description unreliable and has to be fixed before anyone can take the numbers seriously.

Minor issues: the abstract says fine-tuning with local data but the experiments describe retraining from scratch; there are no error bars or significance tests; and the IQA model is trained on synthetic distortions, so its application to PAD filtering needs external validation. None of those are fatal, but they add up.

If the lesion-aware split and the dataset arithmetic are corrected, this paper would be a decent contribution to the clinical deployment literature. As it stands, the evidence for the main claim is not yet trustworthy. I would send it to peer review—the core idea is worth refereeing—but with a clear request for major revisions.

Recommended action: engage with it, but treat the empirical results as provisional until the split and stats are clarified.

Referee Report

4 major / 3 minor

Summary. The paper introduces DermAI, a smartphone-based acquisition platform for clinical dermatology images that performs real-time quality checks, structured metadata capture, and on-device classification. The authors collected a new clinical dataset (3,401 images, 200 lesions) at a Brazilian public university hospital and benchmarked several lightweight CNNs across PAD-UFES-20, DDI, and DermAI. The central claim is that models trained solely on public data fail to generalize to the DermAI acquisition distribution, while training/fine-tuning on DermAI local data substantially improves cross-dataset performance, especially on DDI. The paper also reports that quality filtering PAD images improves accuracy.

Significance. If the dataset description is accurate and the evaluation protocol is correct, the contribution is valuable: it demonstrates that standardized on-device acquisition combined with local data can yield classifiers that transfer better across heterogeneous clinical datasets than models trained on existing public data. The cross-dataset comparison (Table 1) is a useful, falsifiable design, and the DermAI application addresses a real need in low-resource clinical settings. The paper also ships the dataset upon request, which is a strength. However, the central generalization claim depends on whether the DermAI split is lesion-exclusive; the current text is ambiguous on this point and the dataset statistics contain an internal inconsistency, both of which must be resolved before the results can be interpreted.

major comments (4)

[Section 4, Table 1] The DermAI evaluation split is not specified as lesion-aware. Section 3.2 reports 3,401 images from 200 uniquely annotated lesions (about 17 images per lesion on average). If the fixed 80/10/10 split is applied at the image level, images of the same lesion appear in both training and test partitions, allowing the model to memorize lesion-specific features. This would inflate the DermAI-column accuracies in Table 1 (e.g., DenseNet (DermAI) ACC 0.8209 vs. DenseNet (PAD) ACC 0.4648) and directly weaken the abstract's claim that public models fail while local fine-tuning improves. Please state explicitly whether the split is performed on unique lesions (or patients), and if it is not, rerun the experiments with a lesion-exclusive split and report both results.
[Section 3.2] The dataset statistics are internally inconsistent. The text states 3,401 images total with 2,273 benign, 608 malignant, and 520 pre-malignant (sum 3,401), but the subclass counts (melanoma 30 + BCC 477 + SCC 142 + nevus 568 + AK 520 + benign keratosis 706 + solar lentigo 285) sum to 2,728. The missing 673 images are unaccounted for, and the benign/malignant/pre-malignant totals do not match the subclasses. This makes the dataset description unreliable for reproducibility and undermines the claimed class balance. Please reconcile the numbers or provide a corrected class breakdown.
[Section 4, Table 1] All results are reported on a single fixed 80/10/10 split with no error bars, confidence intervals, or statistical significance tests. This matters particularly for the DDI comparison, where the mean ACC across DermAI-trained models (0.4150) is much higher than for PAD-trained models (0.1673), but individual models vary widely (e.g., MobileNet v3 DermAI DDI ACC 0.2927 vs. Resnet DermAI DDI ACC 0.5050). Without repeated splits or bootstrap intervals, the reader cannot assess whether the observed improvements are stable or partly driven by the specific split. Please add variance estimates or cross-validation results for the main comparisons.
[Section 3.1, Table 1, PAD filter rows] The quality-filter claim relies on an IQA model trained on synthetic distortions, but the manuscript does not specify the filtering thresholds, the proportion of PAD images removed, or whether the filter was applied per-image or at the dataset level. The PAD-filtered average in Table 1 (ACC 0.5196 on DermAI) is only slightly above the unfiltered PAD average (0.4975), and on DDI it is slightly lower (0.1583 vs. 0.1673). Please provide the filter's operating point, its validation against human ratings, and the exact composition of the filtered subset so the improvement is interpretable and reproducible.

minor comments (3)

[Throughout] Typographical issues: 'ethinicity' in the abstract, 'Preliminar' in Section 3.3, 'Commitee' in Section 6, and 'A VG-all models' in Table 1 should be corrected.
[Figure 2] The figure label 'DermaiPublic clinical data' is unclear; it appears to be a concatenation of two labels. Please separate into distinct captions or legend entries.
[Section 3.2] The definition of 'uniquely annotated lesions' should be clarified: does it mean one lesion per patient, or one lesion per skin location? This distinction affects the interpretation of the number of patients and the independence of samples.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; external PAD/DDI benchmarks support the central generalization claim, though the unreported DermAI split is a validity caveat.

full rationale

The paper's central claim is an empirical generalization comparison, not a formal derivation: models trained on public datasets transfer poorly to DermAI, while models retrained on DermAI transfer better. The evidence does not reduce to the training inputs by construction. PAD-trained models are evaluated on an 80/10/10 PAD split and on two external sets (DDI and DermAI); DermAI-trained models are evaluated on PAD and DDI as well as on DermAI. The cross-dataset numbers (e.g., DenseNet trained on DermAI evaluated on DDI has F1 0.3103 versus DenseNet trained on PAD evaluated on DDI has F1 0.1537) provide independent support for the abstract's claim. The DermAI quality filter is trained on synthetic distortions and applied to PAD as a curation step; its inputs are not the target performance numbers, so there is no fitted-input-called-prediction cycle. The only author-overlap citation is PAD-UFES-20 [5], a public dataset coauthored by a current author; it is a real, externally testable dataset, not an unverified uniqueness theorem, and is not used to forbid alternatives. A caveat worth noting, but not a circularity: Section 4 specifies 80/10/10 splits only for PAD, and Section 3.2 reports 3,401 images corresponding to 200 unique lesions, so the DermAI internal accuracy in Table 1 could be inflated if the DermAI split is image-level rather than lesion-aware. This is a missing methodological detail and a dataset-validity risk, not a definitional reduction; the external DDI/PAD evaluations remain independent evidence for the central claim. Therefore, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's central claims rest on the quality-filtering pipeline and the dataset split; both involve hand-chosen thresholds and untested assumptions about transfer of synthetic distortions and label reliability.

free parameters (3)

IQA quality thresholds = not specified
The quality assessment model outputs four binary indicators; the threshold at which users are prompted to recapture is hand-chosen and used to filter PAD, affecting results.
Central crop size = not specified
Preprocessing crops a central region and enforces square aspect ratio; crop size is a design choice affecting model input.
Train/validation/test split ratio = 80/10/10
Chosen by hand and applied to PAD and likely DermAI; not justified and can be affected by lesion-level leakage.

assumptions (3)

domain assumption IQA model trained on synthetic distortions transfers to real-world smartphone images
Used to filter PAD; if the synthetic distortions don't match real acquisition artifacts, the filter quality claim fails. Section 3.1.
domain assumption Clinician-provided labels are ground truth
Diagnoses are clinical judgment with histopathology only for suspicious cases; label noise is not quantified. Section 3.1.
domain assumption Accuracy/F1 on DDI and DermAI is a valid proxy for real-world generalization
DDI is a distinct small dataset and DermAI is from the same acquisition pipeline, so generalization to independent clinical populations is not established. Table 1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DermAI: Clinical dermatology acquisition through quality-driven image collection for AI classification in mobile." pith.science (2026). https://pith.science/paper/JODX3CIO

@misc{pith2026251110367,
  author       = {Pith},
  title        = {Pith review of: DermAI: Clinical dermatology acquisition through quality-driven image collection for AI classification in mobile},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JODX3CIO}},
  note         = {Machine review of arXiv:2511.10367}
}

read the original abstract

AI-based dermatology adoption remains limited by biased datasets, variable image quality, and limited validation. We introduce DermAI, a lightweight, smartphone-based application that enables real-time capture, annotation, and classification of skin lesions during routine consultations. Unlike prior dermoscopy-focused tools, DermAI performs on-device quality checks, and local model adaptation. The DermAI clinical dataset, encompasses a wide range of skin tones, ethinicity and source devices. In preliminary experiments, models trained on public datasets failed to generalize to our samples, while fine-tuning with local data improved performance. These results highlight the importance of standardized, diverse data collection aligned with healthcare needs and oriented to machine learning development.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 1 linked inside Pith

[1]

A survey, review, and future trends of skin lesion seg- mentation and classification,

M. K. Hasan, M. A. Ahamad, C. H. Yap, and G. Yang, “A survey, review, and future trends of skin lesion seg- mentation and classification,”Computers in Biology and Medicine, p. 106 624, 2023

2023
[2]

A multimodal vision foundation model for clinical dermatology,

S. Yan et al., “A multimodal vision foundation model for clinical dermatology,”Nature Medicine, pp. 1–12, 2025

2025
[3]

Mm-skin: Enhancing dermatology vision-language model with an image-text dataset de- rived from textbooks,

W. Zeng et al., “Mm-skin: Enhancing dermatology vision-language model with an image-text dataset de- rived from textbooks,”arXiv preprint arXiv:2505.06152, 2025

arXiv 2025
[4]

Disparities in dermatology AI per- formance on a diverse, curated clinical image set,

R. Daneshjou et al., “Disparities in dermatology AI per- formance on a diverse, curated clinical image set,” en, Sci. Adv., vol. 8, no. 32, eabq6147, Aug. 2022

2022
[5]

PAD-UFES-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones,

A. G. C. Pacheco et al., “PAD-UFES-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones,” en,Data Brief, vol. 32, no. 106221, p. 106 221, Oct. 2020

2020
[6]

The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,

P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,”Scientific data, vol. 5, no. 1, pp. 1–9, 2018

2018
[7]

Lack of transparency and potential bias in artificial intelligence data sets and algorithms: A scoping review,

R. Daneshjou et al., “Lack of transparency and potential bias in artificial intelligence data sets and algorithms: A scoping review,”JAMA dermatology, vol. 157, no. 11, pp. 1362–1369, 2021

2021
[8]

How medical ai devices are evaluated: Limitations and recommendations from an analysis of fda approvals,

E. Wu et al., “How medical ai devices are evaluated: Limitations and recommendations from an analysis of fda approvals,”Nature Medicine, vol. 27, no. 4, pp. 582– 584, 2021

2021

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

A survey, review, and future trends of skin lesion seg- mentation and classification,

M. K. Hasan, M. A. Ahamad, C. H. Yap, and G. Yang, “A survey, review, and future trends of skin lesion seg- mentation and classification,”Computers in Biology and Medicine, p. 106 624, 2023

2023

[2] [2]

A multimodal vision foundation model for clinical dermatology,

S. Yan et al., “A multimodal vision foundation model for clinical dermatology,”Nature Medicine, pp. 1–12, 2025

2025

[3] [3]

Mm-skin: Enhancing dermatology vision-language model with an image-text dataset de- rived from textbooks,

W. Zeng et al., “Mm-skin: Enhancing dermatology vision-language model with an image-text dataset de- rived from textbooks,”arXiv preprint arXiv:2505.06152, 2025

arXiv 2025

[4] [4]

Disparities in dermatology AI per- formance on a diverse, curated clinical image set,

R. Daneshjou et al., “Disparities in dermatology AI per- formance on a diverse, curated clinical image set,” en, Sci. Adv., vol. 8, no. 32, eabq6147, Aug. 2022

2022

[5] [5]

PAD-UFES-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones,

A. G. C. Pacheco et al., “PAD-UFES-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones,” en,Data Brief, vol. 32, no. 106221, p. 106 221, Oct. 2020

2020

[6] [6]

The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,

P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,”Scientific data, vol. 5, no. 1, pp. 1–9, 2018

2018

[7] [7]

Lack of transparency and potential bias in artificial intelligence data sets and algorithms: A scoping review,

R. Daneshjou et al., “Lack of transparency and potential bias in artificial intelligence data sets and algorithms: A scoping review,”JAMA dermatology, vol. 157, no. 11, pp. 1362–1369, 2021

2021

[8] [8]

How medical ai devices are evaluated: Limitations and recommendations from an analysis of fda approvals,

E. Wu et al., “How medical ai devices are evaluated: Limitations and recommendations from an analysis of fda approvals,”Nature Medicine, vol. 27, no. 4, pp. 582– 584, 2021

2021