Divergent Realities: A Comparative Analysis of Human Expert vs. Artificial Intelligence Based Generation and Evaluation of Treatment Plans in Dermatology

Dipayan Sengupta; Saumya Panda

REVIEW 3 major objections 6 minor 12 references

Divergent Realities: A Comparative Analysis of Human Expert vs. Artificial Intelligence Based Generation and Evaluation of Treatment Plans in Dermatology

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The 'better' treatment plan depends on who evaluates it.

desk verdict A suggestive but statistically overclaimed demonstration of an evaluator effect in dermatology treatment plans; the core phenomenon may be real, but the headline p-values only hold under one-sided tests on five cases, so the paper needs revision, not rejection. read the letter →

arxiv 2507.05716 v1 pith:PY3PMD6M submitted 2025-07-08 cs.AI

Dipayan Sengupta , Saumya Panda This is my paper

classification cs.AI

keywords evaluatoreffecttreatmentplansdermatologylargelanguagemodelsGPT-4oo3Gemini2.5Prohuman-AIcomparison

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the perceived quality of a clinical treatment plan depends fundamentally on the nature of the evaluator. In a two-phase experiment, ten dermatologists, GPT-4o, and the reasoning model o3 each generated treatment plans for five complex cases; human experts then scored the plans, and so did the Gemini 2.5 Pro model using the same rubric. Human scorers gave significantly higher marks to human-written plans (7.62 vs. 7.16, p=0.0313), while the AI judge gave significantly higher marks to AI-written plans (7.75 vs. 6.79, p=0.0313), completely inverting the rankings. The authors argue this 'evaluator effect' means neither humans nor AI can serve as an unbiased benchmark for clinical planning, and that explainable human-in-the-loop systems are the necessary path forward.

What carries the argument

The load-bearing mechanism is the dual-phase evaluation design: 60 anonymized, text-normalized treatment plans (5 cases times 12 authors) scored first by ten human dermatologists and then by the Gemini 2.5 Pro model, using an identical holistic 0-10 rubric covering efficacy, safety, patient-centeredness, feasibility, and appropriateness. Wilcoxon signed-rank tests, a linear mixed-effects model, and intraclass correlation quantify the divergence and control for case and rater variability. The central object is the 'evaluator effect': the measured finding that plan rankings flip systematically with the nature of the evaluator, making the evaluator itself a variable in any quality assessment.

What would settle it

Have an independent panel of dermatologists from outside the study plus several AI judges from different model families score the same plans against a checklist derived from clinical guidelines, and ideally compare the plans against real patient outcomes. If human and AI evaluators converge on the same top plans once stylistic confounds are removed, the evaluator effect would be shown to be an artifact of the particular human cohort and the single AI judge rather than a general property of plan quality.

Watch

Extended reading notes

Core claim

The central discovery is a complete inversion of treatment-plan rankings when the evaluator switches from human to AI. Human dermatologists awarded peer-written plans a mean score of 7.62 versus 7.16 for AI plans (Wilcoxon signed-rank p=0.0313), ranking GPT-4o 6th and o3 11th. The Gemini 2.5 Pro judge, using the identical rubric, awarded AI plans 7.75 versus 6.79 for human plans (p=0.0313), ranking o3 first (mean 8.20) and GPT-4o second, with all ten human experts below them. The authors interpret this as a gap between experience-based clinical heuristics and data-driven algorithmic logic, epitomized by what they call the 'o3 paradox': the same plan that clinicians rated nearly last was rated first by a sophisticated AI.

Load-bearing premise

The single Gemini 2.5 Pro run is treated as a 'superior' and unbiased judge, with no calibration against clinical outcomes or independent ground truth, so its preference for AI plans might reflect LLM self-preference, style, or stochastic noise rather than data-driven optimality.

Editorial extensions

If this is right

If true, any single-evaluator benchmark of AI treatment plans is incomplete; results must be reported separately for human and AI judges.
A reasoning model like o3 can be ranked at the bottom by clinicians and at the top by an AI judge, so its clinical value cannot be settled by either group alone.
Human-in-the-loop decision support with explicit rationales, i.e., explainable AI, becomes necessary to bridge the gap between algorithmic recommendations and clinical acceptance.
The 'o3 paradox' implies AI may propose evidence-supported but unfamiliar plans that localized expert cohorts will systematically undervalue, hindering adoption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The inversion may partly reflect LLM self-preference or style matching, such as Gemini recognizing textual patterns typical of other AI models, rather than purely evidence-based quality; a testable extension is to run several AI judges from different model families and multiple runs to measure variance and detect systematic self-bias.
The GPT-4-based normalization step may have shifted all plans toward a common stylistic register, potentially affecting human and AI ratings differently, for example by erasing human voice or adding AI-like phrasing.
A decisive test would be an independent panel of international experts or real patient outcomes adjudicating the same plans; if they agree with one evaluator class, the evaluator effect is an artifact of that class's preferences, not a property of plan quality.
Plan content could be scored against guideline checklists to separate objective treatment adherence from holistic perceived quality, isolating what each evaluator type actually rewards.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core observation here is worth taking seriously: human dermatologists scored human plans higher than AI plans, and one AI judge scored AI plans higher than human plans, with o3 flipping from rank 11 to rank 1. That is a clean, provocative demonstration of an evaluator effect in a clinical planning context, and I don't think the effect is an artifact of the scoring tables. The authors also did some things right: they normalized responses to control for style, blinded the human raters, used a mixed-effects model for the human phase, and reported ICC. The idea that AI-generated plans may be systematically undervalued by human experts because of unfamiliarity, not because they are worse, is a genuinely important hypothesis for clinical AI deployment.

The soft spots are real and they are mostly statistical. The reported p=0.0313 for both phases is exactly the one-sided Wilcoxon p-value for n=5 pairs; the two-sided p is 0.0625, which is not significant. The Methods never state a one-sided alternative, so the claim of a 'statistically significant' evaluator effect is not supported as written. The linear mixed-effects model in Phase 1 gives p=0.012 for o3 being scored lower, so the human-phase result has some independent support, but there is no equivalent model for Phase 2, which rests on a single unrepeated Gemini run. Calling Gemini a 'superior AI judge' and interpreting its preferences as 'evidence-based optimality' is an assumption, not a finding, especially without any external gold standard or clinical outcome data. The missing supplementary materials (case texts, prompts, raw scores) also make the experiment impossible to reproduce from the arXiv version.

The framing overreaches: 'fundamentally dependent on the evaluator' and 'deep gap in reasoning' go beyond what five cases and one AI judge can show. But I want to be clear that the underlying question is legitimate and the study is a reasonable empirical probe, not a toy or a disguised simulation. The authors acknowledge the small sample and the lack of objective precision, which is honest.

Who is this for? Researchers working on LLM-as-judge in medical domains, and anyone thinking about how to validate AI-generated clinical plans. It deserves a serious referee, because the topic matters and the empirical setup is a real extension of prior work. But it needs major revisions: share the raw data, repeat the AI judge evaluation with multiple runs and report variance, use two-sided tests or pre-specified one-sided hypotheses, and temper the conclusions to match the evidentiary weight. If the authors do that, this could be a useful cautionary data point.

Referee Report

3 major / 6 minor

Summary. The manuscript reports a two-phase comparative study in which 12 'participants' (10 board-certified dermatologists, GPT-4o, and o3) produced treatment plans for five hypothetical complex dermatology cases. After an anonymization/normalization step, all 60 plans were scored on a 0–10 rubric first by the 10 human experts (Phase 1) and then by a single run of Gemini 2.5 Pro (Phase 2). The authors report that human evaluators scored human-authored plans higher than AI-authored plans (7.62 vs 7.16, p=0.0313) while the AI judge scored AI plans higher (7.75 vs 6.79, p=0.0313), with o3 moving from rank 11 to rank 1. They interpret this as evidence of a fundamental 'evaluator effect' and a gap between experience-based human heuristics and data-driven algorithmic optimality.

Significance. The question is timely and the design has real strengths: the paired structure compares the same plans under the same rubric, raters were blinded to source, and the authors commit to releasing disaggregated scores. If the evaluator effect is genuine, it matters for any clinical AI evaluation pipeline and for LLM-as-judge methodology in medicine. However, the statistical and interpretive support is currently weaker than the abstract claims: the main p-values correspond to one-sided tests on five cases, the AI judge is a single unrepeated sample, and there is no external standard of plan quality. The paper's value is therefore as a descriptive, hypothesis-generating observation rather than as a demonstration of 'profound' or 'fundamental' divergence.

major comments (3)

[Results, Phase 1 and Phase 2; Methods, Statistical Analysis] The reported p=0.0313 for both Phase 1 and Phase 2 is the exact one-sided Wilcoxon signed-rank p-value for five paired observations (0.03125), while the corresponding two-sided p-value is 0.0625. The Methods never state that a one-sided alternative was pre-specified, and reporting the same p-value for opposite directional claims in the two phases is only possible if separate one-sided tests were used. With n=5 cases, the two-sided test cannot reach 0.05, so the claim of a 'profound and statistically significant evaluator effect' (Abstract; Discussion) is not supported as stated. The unit of analysis (case-level means vs. pooled evaluations) is also not specified; the aggregate 450-vs-100 comparison would violate independence if tested as pooled scores. The Phase 1 linear mixed-effects model is stronger evidence for the o3-specific effect, but no equivalent model is reported for Phase 2.
[Methods, Phase 2] Phase 2 rests on a single Gemini 2.5 Pro run: no temperature setting, no repeated sampling, and no measure of run-to-run variability are reported. LLM judges are known to be stochastic and prompt-sensitive (cf. Zheng et al. [6]); the point estimate that o3 ranks first (mean 8.20) could change substantially across runs. Without repeated sampling or at minimum a sensitivity analysis, the 'complete inversion' claim is not established at the level of statistical confidence implied by the paper.
[Discussion, 'o3 paradox'] The interpretive conclusion that Gemini's preference reflects 'the evidence-based optimality of these plans' is an assumption, not a finding. No external gold standard—clinical outcome data, an independent expert panel adjudication, or a validated quality instrument—is used to calibrate either human or AI scores. The manuscript itself labels Gemini a 'superior AI judge' (Methods) without support. The observed divergence in scores is descriptive and robust in direction, but the value judgment that o3's plans are superior, or that human experts are biased, is not derivable from these data.

minor comments (6)

[Table B] In Table B, the source ID for rank 11 is broken across lines as 'uid_ss84Ly7 0' and should read 'uid_ss84Ly70' to match Table A.
[Discussion] The sentence listing cognitive biases contains an apparent stray fragment 'a status quo bias, showing resistance to novel methods outcome)' that should be rephrased.
[Results, Phase 1 and Phase 2] The abstract and Results state p=0.0313 but do not report the test statistic (W) or the sample size (n=5 cases) in the main text; these should be added for transparency.
[Supplementary Materials] The Supplementary Materials are described as 'will be available in the final published version'; since the case vignettes and scoring instructions are central to reproducibility, they should be made available to reviewers or the manuscript should summarize them in an appendix.
[References] Reference 13 is missing the closing bracket before 'This study'.
[Methods, Phase 2] The phrase 'Gemini 2.5 Pro' appears with inconsistent capitalization ('gemini-2.5-pro-preview-05-06' vs 'Gemini 2.5 pro') in Methods and Discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluator-effect finding is an independent empirical measurement, not a derivation that reduces to its inputs.

full rationale

The paper's central observation is a direct empirical comparison: human experts scored the same normalized treatment plans (7.62 vs 7.16) and a separate Gemini 2.5 Pro run scored the same plans in the opposite direction (7.75 vs 6.79). These score tables and the resulting rank inversion are independent measurements of evaluator behavior; the divergence is not constructed from the inputs by definition. The label 'superior AI judge' is an interpretive assumption about Gemini's authority, not a derived consequence, and the discussion's claim that Gemini 'would recognize the evidence-based optimality' of o3's plans is an unvalidated interpretive gloss rather than a circular reduction. There are no fitted parameters renamed as predictions, no self-citations carrying the argument, no imported uniqueness theorems, and no ansatz smuggled in through prior work. The statistical concern about the n=5 Wilcoxon test and the one-sided p=0.0313 is a robustness/correctness issue, not a circularity issue. Therefore, under the stated hard rules, no specific circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters are fitted. The paper's main hidden assumptions are about evaluator validity: that the AI judge is a reliable arbiter, that normalization is content-preserving, and that holistic rubric scores measure clinical quality. These assumptions are load-bearing for the interpretive conclusion, not for the raw score difference.

assumptions (4)

domain assumption Gemini 2.5 Pro is a 'superior' AI judge whose scores are a meaningful measure of treatment plan quality.
Methods Phase 2 labels the model superior and treats its preferences as evidence of algorithmic optimality; no validation against clinical outcomes or expert consensus is provided.
domain assumption The GPT-4-based normalization step preserves all core clinical information and does not systematically bias later evaluation.
Methods Response Normalization claims blinded two-step normalization with manual review, but the preprint provides no comparison of original versus normalized scores.
domain assumption A single holistic 0-10 score based on efficacy, safety, patient-centeredness, feasibility, and appropriateness is a valid measure of clinical plan quality for both evaluator types.
Methods Evaluation Protocol defines the rubric but provides no construct validation or evidence that scores correlate with patient-relevant outcomes.
domain assumption The five author-written case vignettes are representative complex dermatology treatment decisions.
Methods Clinical Case Scenarios states the cases were designed by the authors; the full texts are not in the preprint, so representativeness cannot be checked.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Divergent Realities: A Comparative Analysis of Human Expert vs. Artificial Intelligence Based Generation and Evaluation of Treatment Plans in Dermatology." pith.science (2026). https://pith.science/paper/PY3PMD6M

@misc{pith2026250705716,
  author       = {Pith},
  title        = {Pith review of: Divergent Realities: A Comparative Analysis of Human Expert vs. Artificial Intelligence Based Generation and Evaluation of Treatment Plans in Dermatology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PY3PMD6M}},
  note         = {Machine review of arXiv:2507.05716}
}

read the original abstract

Background: Evaluating AI-generated treatment plans is a key challenge as AI expands beyond diagnostics, especially with new reasoning models. This study compares plans from human experts and two AI models (a generalist and a reasoner), assessed by both human peers and a superior AI judge. Methods: Ten dermatologists, a generalist AI (GPT-4o), and a reasoning AI (o3) generated treatment plans for five complex dermatology cases. The anonymized, normalized plans were scored in two phases: 1) by the ten human experts, and 2) by a superior AI judge (Gemini 2.5 Pro) using an identical rubric. Results: A profound 'evaluator effect' was observed. Human experts scored peer-generated plans significantly higher than AI plans (mean 7.62 vs. 7.16; p=0.0313), ranking GPT-4o 6th (mean 7.38) and the reasoning model, o3, 11th (mean 6.97). Conversely, the AI judge produced a complete inversion, scoring AI plans significantly higher than human plans (mean 7.75 vs. 6.79; p=0.0313). It ranked o3 1st (mean 8.20) and GPT-4o 2nd, placing all human experts lower. Conclusions: The perceived quality of a clinical plan is fundamentally dependent on the evaluator's nature. An advanced reasoning AI, ranked poorly by human experts, was judged as superior by a sophisticated AI, revealing a deep gap between experience-based clinical heuristics and data-driven algorithmic logic. This paradox presents a critical challenge for AI integration, suggesting the future requires synergistic, explainable human-AI systems that bridge this reasoning gap to augment clinical care.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

[6]

Judging llm-as-a-judge with mt-bench and chatbot arena

Zheng L, Chiang WL, Sheng Y, Zhuang S, Wu Z, Zhuang Y, Lin Z, Li Z, Li D, Xing E, Zhang H. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems. 2023 Dec 15;36:46595-623

work page 2023
[1]

Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda

Kumar Y, Koul A, Singla R, Ijaz MF. Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda. Journal of ambient intelligence and humanized computing. 2023 Jul;14(7):8459-86

work page 2023
[2]

How Doctors Think

Mongtomery K. How Doctors Think. Oxford University Press; 2005. 3. Bieber T, Nestle F. Personalized Treatment Options in Dermatology. Springer; 2015

work page 2005
[4]

GPT-4 is here: what scientists think

Sanderson K. GPT-4 is here: what scientists think. Nature. 2023 Mar 30;615(7954):773

work page 2023
[5]

OpenAI o3 and o4-mini System Card OpenAI [Internet]. 2025. Available from: https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf

work page 2025
[7]

Cognitive biases and heuristics in medical decision making: a critical review using a systematic search strategy

Blumenthal-Barby JS, Krieger H. Cognitive biases and heuristics in medical decision making: a critical review using a systematic search strategy. Medical decision making. 2015 May;35(4):539-57

work page 2015
[8]

Datasets for large language models: A comprehensive survey

Liu Y, Cao J, Liu C, Ding K, Jin L. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18041. 2024 Feb 28

arXiv 2024
[9]

Is AI the future of evaluation in medical education?? AI vs

Tekin M, Yurdal MO, Toraman Ç, Korkmaz G, Uysal İ. Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination. BMC medical education. 2025 May 1;25(1):641

work page 2025

Show all 12 references

[10]

AI in health: keeping the human in the loop

Bakken S. AI in health: keeping the human in the loop. Journal of the American Medical Informatics Association. 2023 Jul 1;30(7):1225-6

2023
[11]

A historical perspective of explainable artificial intelligence

Confalonieri R, Coba L, Wagner B, Besold TR. A historical perspective of explainable artificial intelligence. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 2021 Jan;11(1):e1391

2021
[12]

A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

Gambetti A, Han Q, Shen H, Soares C. A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems. arXiv preprint arXiv:2502.09849. 2025 Feb 14

2025
[13]

Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)

Adadi A, Berrada M. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access. 2018 Sep 16;6:52138-60

2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [6]

Judging llm-as-a-judge with mt-bench and chatbot arena

Zheng L, Chiang WL, Sheng Y, Zhuang S, Wu Z, Zhuang Y, Lin Z, Li Z, Li D, Xing E, Zhang H. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems. 2023 Dec 15;36:46595-623

work page 2023

[2] [1]

Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda

Kumar Y, Koul A, Singla R, Ijaz MF. Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda. Journal of ambient intelligence and humanized computing. 2023 Jul;14(7):8459-86

work page 2023

[3] [2]

How Doctors Think

Mongtomery K. How Doctors Think. Oxford University Press; 2005. 3. Bieber T, Nestle F. Personalized Treatment Options in Dermatology. Springer; 2015

work page 2005

[4] [4]

GPT-4 is here: what scientists think

Sanderson K. GPT-4 is here: what scientists think. Nature. 2023 Mar 30;615(7954):773

work page 2023

[5] [5]

OpenAI o3 and o4-mini System Card OpenAI [Internet]. 2025. Available from: https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf

work page 2025

[6] [7]

Cognitive biases and heuristics in medical decision making: a critical review using a systematic search strategy

Blumenthal-Barby JS, Krieger H. Cognitive biases and heuristics in medical decision making: a critical review using a systematic search strategy. Medical decision making. 2015 May;35(4):539-57

work page 2015

[7] [8]

Datasets for large language models: A comprehensive survey

Liu Y, Cao J, Liu C, Ding K, Jin L. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18041. 2024 Feb 28

arXiv 2024

[8] [9]

Is AI the future of evaluation in medical education?? AI vs

Tekin M, Yurdal MO, Toraman Ç, Korkmaz G, Uysal İ. Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination. BMC medical education. 2025 May 1;25(1):641

work page 2025