pith. sign in

arxiv: 2606.07342 · v1 · pith:5DRPSEBEnew · submitted 2026-06-05 · 💻 cs.CL · cs.NE

LLM-Guided Evolution for Medical Decision Pipelines

Ivan Sviridov , Artem Oskin , Ivan Panin , Iaroslav Bespalov , Dmitry Dylov , Ivan Oseledets , Aleksandr Nesterov This is my paper

Pith reviewed 2026-06-27 22:12 UTC · model grok-4.3

classification 💻 cs.CL cs.NE
keywords LLM-guided evolutionMAP-Elitesmedical triageinteractive consultationmedical image classificationevolutionary searchclinical decision supportexecutable programs
0
0 comments X

The pith

LLM-guided evolutionary search over executable programs outperforms hand-designed baselines in medical triage, consultation, and image classification.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether large language models can steer an evolutionary algorithm to automatically discover effective medical decision strategies at inference time, without fine-tuning or extensive manual engineering. It casts urgency triage, interactive consultation, and medical image classification as searches over executable artifacts that are scored by task-specific fitness functions. Across the three settings the evolved artifacts improve accuracy, recall, safety-weighted performance, and efficiency frontiers relative to manually designed baselines, with gains visible on held-out data. A reader would care because the approach supplies a practical route to adapt language models to clinical workflows while keeping outputs interpretable and format-constrained. The reported advantages trace to changes in decision logic such as calibrated thresholds and selective evidence gathering rather than surface-level rewording.

Core claim

Formulating medical decision tasks as LLM-guided MAP-Elites evolutionary searches over executable programs or prompts, optimized by task-specific fitness functions, produces artifacts that raise Semigran accuracy from 77.3 percent to 87.1 percent and emergency recall from 0.60 to 0.97 in triage, improve the accuracy-cost frontier across multiple base models in consultation with transfer to held-out data, and lift frozen vision-language model performance on PneumoniaMNIST while preserving strict JSON output formats.

What carries the argument

LLM-guided MAP-Elites evolution of executable artifacts scored by task-specific fitness functions.

If this is right

  • Triage programs achieve higher accuracy and recall while also improving safety-weighted performance on held-out MIMIC-ESI data.
  • Consultation policies raise the accuracy-cost frontier for Llama-3, Qwen-3.5, and Gemma-4 and transfer to held-out iCRAFTMD.
  • Prompt-only evolution improves frozen MedGemma performance on PneumoniaMNIST without altering required JSON output structure.
  • Qualitative gains arise from interpretable mechanisms such as calibrated triage boundaries, targeted evidence acquisition, and selective commitment rather than prompt rewording alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same search process could be rerun periodically on fresh patient data to keep decision pipelines aligned with changing guidelines or populations.
  • Because the evolved artifacts remain executable and human-readable, they could be audited or edited by clinicians before deployment.
  • The approach may generalize to other sequential decision domains that already use promptable models and can supply scalar fitness signals.
  • Success depends on designing fitness functions that penalize unsafe or biased behavior; poor choices could lock in hidden errors.

Load-bearing premise

The fitness functions used during evolution accurately reflect real clinical utility, safety, and generalization to new data without rewarding superficial patterns.

What would settle it

When the evolved programs are run on a large independent clinical dataset never seen during evolution, their accuracy, recall, and safety metrics fall to or below those of the manual baselines.

Figures

Figures reproduced from arXiv: 2606.07342 by Aleksandr Nesterov, Artem Oskin, Dmitry Dylov, Iaroslav Bespalov, Ivan Oseledets, Ivan Panin, Ivan Sviridov.

Figure 1
Figure 1. Figure 1: Semigran MAP-Elites fitness dynamics over [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MIMIC-ESI evolution dynamics. The plot shows rolling mean archive fitness, median fitness, and the best-so-far frontier over generations. Algorithm Acc (%) ↑ Avg. tok. ↓ Rank ↑ Llama-3-8B Scale 56.2 ± 1.2 2727 ± 10 2 + 0 = 2 Implicit 43.6 ± 4.3 154 ± 2 0 + 4 = 4 Binary 57.4 ± 0.4 2137 ± 18 1 + 1 = 2 Numerical cutoff 56.4 ± 0.7 3463 ± 20 3 + 2 = 5 Evolved 63.8 ± 1.1 156 ± 13 4+3=7 Llama-3-70B Scale 75.7 ± 1… view at source ↗
read the original abstract

Adapting large language models (LLMs) to clinical workflows often requires costly fine-tuning or manual prompt and pipeline engineering. We study LLM-guided MAP-Elites evolution as an inference-time alternative for discovering medical decision strategies and provide an implementation repository at https://github.com/univanxx/llm_guided_evo_medical. We formulate urgency triage, interactive consultation, and medical image classification as evolutionary searches over executable artifacts optimized by task-specific fitness functions. Across all three settings, evolution improves over manually designed baselines under practical constraints. In triage, evolved programs increase Semigran accuracy from $77.3\%$ to $87.1\%$ and emergency recall from $0.60$ to $0.97$, while improving safety-weighted held-out MIMIC-ESI performance. In interactive consultation, evolved policies improve the accuracy--cost frontier across Llama-3, Qwen-3.5, and Gemma-4 and transfer to held-out iCRAFTMD. In PneumoniaMNIST, prompt-only evolution improves frozen MedGemma VLMs while preserving strict JSON outputs. Qualitative analysis shows that the gains come from interpretable program-level mechanisms, calibrated triage boundaries, targeted evidence acquisition, selective commitment, and finding-oriented visual decision rules, rather than superficial prompt rewording alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes LLM-guided MAP-Elites evolutionary search as an inference-time method to discover executable programs for three medical decision tasks: urgency triage, interactive consultation, and medical image classification. It reports that evolved programs outperform manually designed baselines, with specific gains such as Semigran accuracy rising from 77.3% to 87.1% and emergency recall from 0.60 to 0.97 in triage, improved accuracy-cost frontiers in consultation across multiple LLMs with transfer to held-out iCRAFTMD, and better performance in PneumoniaMNIST while preserving JSON output constraints. The approach emphasizes interpretable program-level mechanisms and provides an open implementation repository.

Significance. If the reported gains prove robust under proper controls for fitness validation and statistical rigor, the work would demonstrate a practical alternative to fine-tuning or manual prompt engineering for clinical LLM pipelines, with potential for interpretable, constraint-preserving adaptations that generalize across models and datasets.

major comments (3)
  1. [Abstract / Results] Abstract and results presentation: The quantitative claims (Semigran accuracy 77.3%→87.1%, emergency recall 0.60→0.97, safety-weighted MIMIC-ESI improvements) are presented without any description of data splits, number of evolutionary runs, statistical testing procedures, or explicit baseline construction details. This absence directly undermines verifiability of the central empirical claim that evolution improves over manual baselines.
  2. [Methods (fitness functions)] Fitness function definition (methods): The task-specific fitness functions that guide MAP-Elites are not shown to have been validated against expert clinical judgment or patient-level outcomes. Without explicit anti-overfitting safeguards (e.g., held-out fitness validation sets distinct from evolution data) or correlation analysis with real utility metrics, the observed held-out gains could arise from exploitation of proxy definitions rather than genuine clinical improvement.
  3. [Results (consultation and triage sections)] Transfer and generalization claims: The reported transfer to held-out iCRAFTMD and safety-weighted MIMIC-ESI performance is stated without accompanying ablation on whether the fitness functions were computed on distributions that overlap with these held-out sets, leaving open the possibility that gains reflect distribution matching rather than robust policy discovery.
minor comments (2)
  1. [Abstract] The manuscript mentions an implementation repository but does not include a direct link or citation in the main text; adding an explicit footnote or reference would improve reproducibility.
  2. [Introduction / Methods] Notation for the three settings (triage, consultation, image classification) could be introduced with consistent subsection headings to aid navigation between the evolutionary formulation and the per-task results.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive comments, which highlight important aspects of experimental transparency and validation. We address each major comment below, indicating where revisions will be made to improve the manuscript.

read point-by-point responses

  1. Referee: [Abstract / Results] Abstract and results presentation: The quantitative claims (Semigran accuracy 77.3%→87.1%, emergency recall 0.60→0.97, safety-weighted MIMIC-ESI improvements) are presented without any description of data splits, number of evolutionary runs, statistical testing procedures, or explicit baseline construction details. This absence directly undermines verifiability of the central empirical claim that evolution improves over manual baselines.

    Authors: We agree that the abstract and main results sections require additional methodological details for full verifiability. The full manuscript and supplementary materials already specify the data splits (e.g., standard train/validation/test partitions for each dataset), the number of independent evolutionary runs (typically 5–10 with reported means and standard deviations), statistical testing (paired comparisons with p-values), and baseline construction (manually designed programs by the authors following clinical guidelines). In the revision we will move these details into the main text of the abstract, results, and methods sections, and add a dedicated experimental setup subsection. revision: yes

  2. Referee: [Methods (fitness functions)] Fitness function definition (methods): The task-specific fitness functions that guide MAP-Elites are not shown to have been validated against expert clinical judgment or patient-level outcomes. Without explicit anti-overfitting safeguards (e.g., held-out fitness validation sets distinct from evolution data) or correlation analysis with real utility metrics, the observed held-out gains could arise from exploitation of proxy definitions rather than genuine clinical improvement.

    Authors: The fitness functions are proxy metrics computed from available ground-truth labels (e.g., urgency categories, consultation accuracy vs. cost, classification accuracy under JSON constraints). We did enforce separation between the data used to compute fitness during evolution and the final held-out test sets. However, we did not conduct new expert clinical validation or patient-outcome correlation studies, which would require resources beyond the scope of this work. In revision we will expand the methods section to explicitly describe the proxy definitions, document the train/test separation used as an anti-overfitting measure, and add a limitations paragraph acknowledging that proxy fitness may not perfectly align with downstream clinical utility. revision: partial

  3. Referee: [Results (consultation and triage sections)] Transfer and generalization claims: The reported transfer to held-out iCRAFTMD and safety-weighted MIMIC-ESI performance is stated without accompanying ablation on whether the fitness functions were computed on distributions that overlap with these held-out sets, leaving open the possibility that gains reflect distribution matching rather than robust policy discovery.

    Authors: Fitness functions for evolution were computed exclusively on the training portions of the respective datasets (Semigran triage data, iCRAFTMD training split, PneumoniaMNIST training split), while the reported transfer results use completely disjoint held-out test sets (held-out iCRAFTMD and MIMIC-ESI). In the revision we will add an explicit ablation paragraph and table confirming the non-overlapping data partitions and showing that performance gains persist on these strictly held-out distributions, thereby addressing the distribution-matching concern. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical comparisons with no derivations

full rationale

The paper reports direct empirical gains from LLM-guided MAP-Elites evolution versus manual baselines across triage, consultation, and image classification tasks, measured on held-out sets (MIMIC-ESI, iCRAFTMD) and via qualitative inspection. No equations, uniqueness theorems, fitted parameters renamed as predictions, or self-citation chains appear in the provided text or abstract. The central claims rest on observable performance deltas rather than any reduction to inputs by construction, satisfying the default expectation of a non-circular empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Review is based on abstract only; the central claim rests on the unexamined premise that the chosen fitness functions are faithful proxies for clinical value and that evolutionary search under LLM guidance discovers non-trivial improvements rather than artifacts of the fitness definition.

axioms (1)
  • domain assumption Task-specific fitness functions can be defined that reliably guide evolution toward clinically useful and safe decision strategies
    The paper states that programs are optimized by task-specific fitness functions but provides no further justification or validation of those functions.

pith-pipeline@v0.9.1-grok · 5788 in / 1273 out tokens · 27476 ms · 2026-06-27T22:12:35.759662+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages

  1. [1]

    Adebayo Da’Costa, Jennifer Teke, Joseph E Origbo, Ayokunle Osonuga, Eghosasere Egbon, and David B Olawade

    Exploration and exploitation in evolution- ary algorithms: A survey.ACM computing surveys (CSUR), 45(3):1–33. Adebayo Da’Costa, Jennifer Teke, Joseph E Origbo, Ayokunle Osonuga, Eghosasere Egbon, and David B Olawade. 2025. Ai-driven triage in emergency de- partments: A review of benefits, challenges, and fu- ture directions.International Journal of Medica...

    work page arXiv 2025

  2. [2]

    Version 2.2

    MIMIC-IV-ED.PhysioNet. Version 2.2. Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Leandra A Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M Van Allen, David Kim, and 1 others. 2025. An eval- uation framework for clinical use of large language models in patient interaction tasks.Nature medicine, 31(1):77–86. Valen...

    work page arXiv 2025

  3. [3]

    Lehman, J

    Evolution through large models.Preprint, arXiv:2206.08896. 10 David M Levine, Rudraksh Tuwani, Benjamin Kompa, Amita Varma, Samuel G. Finlayson, Ateev Mehrotra, and Andrew Beam. 2023. The diagnostic and triage accuracy of the gpt-3 artificial intelligence model. medRxiv. Shuyue Stella Li, Jimin Mun, Faeze Brahman, Pedram Hosseini, Bryceton G Thomas, Jessi...

    work page arXiv 2023

  4. [4]

    In The Eleventh International Conference on Learning Representations

    Medical image understanding with pretrained vision language models: A comprehensive study. In The Eleventh International Conference on Learning Representations. Iain Robertson-Steel. 2006. Evolution of triage systems. Emergency medicine journal, 23(2):154–155. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kuma...

    work page arXiv 2006

  5. [5]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26625–26665

    3mdbench: Medical multimodal multi-agent dialogue benchmark. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26625–26665. 11 Qwen Team. 2026. Qwen3.5: Accelerating productivity with native multimodal agents. Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li...

    2025

  6. [6]

    arXiv preprint arXiv:2502.15944

    AutoMedPrompt: A new framework for opti- mizing LLM medical prompts using textual gradients. arXiv preprint arXiv:2502.15944. Yuchen Xia, Jiho Kim, Yuhan Chen, Haojie Ye, Souvik Kundu, Cong Callie Hao, and Nishil Talati. 2024. Understanding the performance and estimating the cost of llm fine-tuning. In2024 IEEE International Symposium on Workload Characte...

    work page arXiv 2024

  7. [7]

    Scientific Data, 10(1):41

    Medmnist v2: A large-scale lightweight bench- mark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41. Yanbo Zhang, Sumeer A Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, and 1 others. 2025. Exploring the role of large language models in the scientific method: from...

    work page arXiv 2025

  8. [8]

    and its public code release1. Here each pro- gram receives a patient description containing his- tory of present illness, demographics, chief com- plaint, and initial vitals, and predicts an ESI level from 1 to 5. This task is more clinically grounded than the vignette benchmark and, crucially, sup- ports a cleaner train/audit/test protocol. We evolve on ...

    2025

  9. [9]

    em": 3,

    and CRAFTMD (Johri et al., 2025) datasets, converted into an interactive setting as iMEDQA and iCRAFTMD using GPT-3.5 (OpenAI, 2026) as the Patient model. To ensure a fair comparison, we follow the original MEDIQ setup by using the same Patient model and evaluating consultation strate- gies by mean answer accuracy across evaluated cases. A.2.2 Candidates ...

    2025

  10. [10]

    Systematically inspect the right and left lung fields for any of the ,→following findings: - focal or lobar airspace opacity - consolidation - patchy infiltrates - air bronchograms - perihilar opacity >=0.5 cm or diffuse subtle opacities

  11. [11]

    pneumonia

    If any such finding is present in either lung, label pneumonia ( ,→class_id 1, class_name "pneumonia")

  12. [12]

    If findings are present in both lungs, choose the dominant pathology ( ,→the lung with the more extensive opacity) - still pneumonia

  13. [13]

    If no findings from step 1 are observed, label normal (class_id 0, ,→class_name "normal")

  14. [14]

    If you are uncertain, select the class you deem most likely based on ,→the visual evidence (do NOT default to normal)

  15. [15]

    The output must be a single line

    Do NOT include any explanations, markdown, or extra characters before ,→or after the JSON. The output must be a single line. """ defentrypoint(): returnPROMPT 22 Task Runs Evolver calls/run Evolver tokens/call Expert calls/eval Expert tokens/call MIMIC-ESI triage (estimated) 1 2,101 in:9,932/11,472/12,143 out:2,468/2,666/3,009 214/347/358 in:327/429/583 o...

    2024