pith. sign in

arxiv: 2605.30295 · v3 · pith:VDLBHV3Fnew · submitted 2026-05-28 · 💻 cs.CL · cs.AI

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

Valentina Bui Muti , Eug\'enie Dulout , Ziquan Fu This is my paper

Pith reviewed 2026-07-04 00:24 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords FHIRLLMsclinical reasoningdiagnostic accuracyelectronic health recordsbenchmarkingsynthetic dataHL7
0
0 comments X

The pith

LLMs achieve lower diagnostic accuracy on structured FHIR bundles than on plain text clinical cases.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a pipeline that converts unstructured diagnostic cases into valid HL7 FHIR R4 bundles through staged LLM generation followed by terminology-grounded validation and repair. Applying the pipeline to MedCaseReasoning produces MedCase-Structured, a dataset of 1,732 synthetic bundles with a 97.1 percent completion rate. Evaluation of multiple LLMs on this dataset shows consistently lower diagnostic accuracy when inputs are the structured FHIR bundles rather than the original plain text. The authors argue that benchmarks aligned with interoperable clinical data formats are required to assess real-world deployment of clinical decision support systems.

Core claim

A reusable pipeline generates terminology-grounded HL7 FHIR R4 bundles from text by combining staged LLM generation with validation and repair to eliminate hallucinated codes and ensure structural and semantic consistency. Applying it to MedCaseReasoning yields MedCase-Structured containing 1,732 complete bundles. Evaluation on the dataset reveals that LLMs exhibit lower diagnostic accuracy on the structured FHIR inputs than on the corresponding plain text cases.

What carries the argument

The staged LLM generation pipeline with terminology-grounded validation and repair that produces complete, valid FHIR bundles from text.

If this is right

  • Structured FHIR formats introduce additional challenges for LLM diagnostic reasoning compared with plain text.
  • Clinical AI benchmarks should incorporate interoperable data formats such as FHIR to match real deployment conditions.
  • The pipeline allows controlled testing of clinical decision support systems on structured inputs.
  • Synthetic but terminology-validated bundles can isolate format-related effects in clinical reasoning evaluations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the accuracy gap holds on real EHR systems, models may require targeted training on structured clinical data representations.
  • The same pipeline could be applied to generate structured test sets for other clinical tasks such as treatment recommendation or outcome prediction.
  • Systematic variation of terminology systems or FHIR resource profiles within the pipeline could identify which structural elements most affect model performance.

Load-bearing premise

The synthetic FHIR bundles serve as realistic proxies for actual electronic health record data such that observed performance differences reflect format effects rather than artifacts of the generation process.

What would settle it

Re-running the same LLMs on real de-identified EHR data converted to FHIR format and checking whether the accuracy drop matches the size observed on the synthetic bundles.

Figures

Figures reproduced from arXiv: 2605.30295 by Eug\'enie Dulout, Valentina Bui Muti, Ziquan Fu.

Figure 1
Figure 1. Figure 1: Overview of MedCase-Structured. (A) Free-text cases are converted into terminology-grounded HL7 FHIR R4 bundles. (B) An example MedCaseReasoning (Wu et al., 2025) case shows extraction, grounding, and rejection of an invalid RxNorm code. (C) Diagnosis-masked bundles are used for EHR-congruent CDSS evaluation against ground-truth diagnosis. isting approaches do not provide flexible and controllable methods … view at source ↗
read the original abstract

Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in structured, electronic health record-congruent settings remains limited. Existing benchmarks often rely on static datasets or unstructured inputs that do not reflect the interoperable data formats used in clinical systems. We introduce a reusable pipeline for generating terminology-grounded HL7 FHIR R4 bundles from unstructured text, enabling controllable evaluation of clinical decision support systems over structured inputs. The pipeline combines staged LLM generation with terminology-grounded validation and repair to eliminate hallucinated codes and enforce structural and semantic consistency. Applying this approach to MedCaseReasoning, we construct MedCase-Structured, a synthetic dataset of 1,732 FHIR bundles derived from clinician-authored diagnostic cases, producing complete, valid bundles for 97.1% of attempted cases. Evaluation on MedCase-Structured reveals consistently lower diagnostic accuracy for LLMs on structured FHIR inputs than with plain text, highlighting the importance of deployment-aligned benchmarking.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces a reusable pipeline that converts unstructured clinician-authored diagnostic cases into terminology-grounded HL7 FHIR R4 bundles via staged LLM generation followed by validation and repair. Applied to MedCaseReasoning, the pipeline yields the MedCase-Structured dataset of 1,732 bundles, reported as 97.1% structurally and semantically valid. Evaluation of LLMs on this dataset shows lower diagnostic accuracy with the structured FHIR inputs than with the original plain-text cases, which the authors interpret as evidence that deployment-aligned benchmarking in realistic EHR formats is needed.

Significance. If the synthetic bundles preserve clinical content and realism without systematic artifacts, the dataset and pipeline would fill a clear gap by enabling controlled evaluation of clinical reasoning systems on interoperable structured data. The reported accuracy gap, if attributable to format rather than generation effects, would strengthen the case for moving beyond text-only benchmarks. The work's main strength is the explicit focus on FHIR R4 validity and the provision of a reusable generation pipeline.

major comments (2)
  1. [Results] Results section (accuracy comparison): the central claim that LLMs exhibit lower diagnostic accuracy on structured FHIR inputs than on plain text is load-bearing for the paper's call for deployment-aligned benchmarking, yet no quantitative comparison of information density, entity coverage, or code-distribution statistics between the original text cases and the generated FHIR bundles is reported. Without such controls, the observed gap cannot be isolated from possible pipeline-induced changes.
  2. [Methods] Methods (pipeline and validation): the staged generation plus terminology repair is presented as ensuring semantic consistency, but the manuscript provides no clinician fidelity ratings or statistical comparison of the synthetic FHIR bundles against authentic EHR FHIR distributions (e.g., code frequency, narrative completeness). This omission directly affects whether performance differences can be attributed to format alone.
minor comments (1)
  1. [Abstract] Abstract: the total number of cases attempted before the 97.1% success rate is not stated, making it difficult to evaluate overall pipeline yield.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and constructive comments, which highlight important considerations for isolating format effects in our evaluation. We have revised the manuscript to incorporate quantitative controls where feasible and to clarify limitations. Below we respond point-by-point to the major comments.

read point-by-point responses

  1. Referee: [Results] Results section (accuracy comparison): the central claim that LLMs exhibit lower diagnostic accuracy on structured FHIR inputs than on plain text is load-bearing for the paper's call for deployment-aligned benchmarking, yet no quantitative comparison of information density, entity coverage, or code-distribution statistics between the original text cases and the generated FHIR bundles is reported. Without such controls, the observed gap cannot be isolated from possible pipeline-induced changes.

    Authors: We agree that explicit controls are necessary to support the interpretation. In the revised manuscript we have added a new analysis subsection (Results 4.3) that reports: (1) entity coverage via clinical NER, showing 91.8% overlap in diagnostic concepts between original cases and FHIR bundles; (2) information density metrics (token counts, code counts, and narrative length); and (3) code-distribution statistics for SNOMED CT, ICD-10, and LOINC, with Kolmogorov-Smirnov tests indicating no significant distributional shift. These additions support that the accuracy gap is attributable to structured format rather than content alteration. We also include representative examples of preserved clinical content. revision: yes

  2. Referee: [Methods] Methods (pipeline and validation): the staged generation plus terminology repair is presented as ensuring semantic consistency, but the manuscript provides no clinician fidelity ratings or statistical comparison of the synthetic FHIR bundles against authentic EHR FHIR distributions (e.g., code frequency, narrative completeness). This omission directly affects whether performance differences can be attributed to format alone.

    Authors: We acknowledge the value of external validation. The revised Methods section now includes statistical comparisons of code frequencies and narrative completeness against publicly available realistic FHIR resources (Synthea-generated bundles and FHIR test servers). These show comparable distributions for common terminologies. Clinician fidelity ratings were not performed in the original study; we have added an explicit limitations paragraph noting this as future work, as it would require a separate IRB-approved human evaluation. The pipeline's 97.1% validity rate and terminology repair step are presented as internal safeguards, with the new controls helping isolate format effects. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical dataset construction and evaluation

full rationale

The paper presents a pipeline for converting text to FHIR bundles and reports LLM evaluation results on the resulting dataset. No mathematical derivations, parameter fitting, predictions, or uniqueness theorems are claimed. All load-bearing steps are explicit data-generation and measurement procedures with no reduction to self-defined quantities or self-citations. The work is self-contained empirical benchmarking.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central contribution rests on domain assumptions about the suitability of LLM generation plus validation for producing realistic FHIR data and the appropriateness of the source MedCaseReasoning cases for conversion.

axioms (2)
  • domain assumption Clinician-authored diagnostic cases from MedCaseReasoning can be converted to valid FHIR bundles using LLM-based generation and validation.
    The pipeline is applied directly to this existing dataset.
  • domain assumption HL7 FHIR R4 is the appropriate standard for representing interoperable clinical data in EHR settings.
    The work targets this format to achieve deployment alignment.

pith-pipeline@v0.9.1-grok · 5708 in / 1226 out tokens · 31306 ms · 2026-07-04T00:24:06.342838+00:00 · methodology

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages · 1 internal anchor

  1. [1]

    doi: 10.1093/jamia/ocad002

    ISSN 1527-974X. doi: 10.1093/jamia/ocad002. URL https://doi.org/10.1093/jamia/ocad002. Frei, J., Feldhus, N., Raithel, L., Roller, R., Meyer, A., and Kramer, F. Infherno: End-to-end Agent-based FHIR Resource Synthesis from Free-form Clinical Notes. In Croce, D., Leidner, J., and Moosavi, N. S. (eds.),Pro- ceedings of the 19th Conference of the European Ch...

    work page doi:10.1093/jamia/ocad002

  2. [2]

    Folio: natural language reasoning with first-order logic, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp

    Association for Computational Linguistics. ISBN 979-8-89176-382-1. doi: 10.18653/v1/ 2026.eacl-demo.13. URL https://aclanthology. org/2026.eacl-demo.13/. HL7 International. FHIR R4 (v4.0.1). URL https:// hl7.org/fhir/R4/index.html. Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B....

    work page doi:10.18653/v1/ 2026

  3. [3]

    doi: 10.1038/ s41597-022-01899-x

    ISSN 2052-4463. doi: 10.1038/ s41597-022-01899-x. URL https://www.nature. com/articles/s41597-022-01899-x. Johnson, J., Douze, M., and J´egou, H. Billion-scale similar- ity search with GPUs, February

    work page 2052

  4. [4]

    Billion-scale similarity search with GPUs

    URL http:// arxiv.org/abs/1702.08734. arXiv:1702.08734 [cs]. Lee, G., Bach, E., Yang, E., Pollard, T., Johnson, A., Choi, E., jia, Y ., and Lee, J. H. FHIR-AgentBench: Bench- marking LLM Agents for Realistic Interoperable EHR Question Answering, November

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    arXiv:2509.19319 [cs]

    URL http:// arxiv.org/abs/2509.19319. arXiv:2509.19319 [cs]. Li, L., Zhou, J., Gao, Z., Hua, W., Fan, L., Yu, H., Hagen, L., Zhang, Y ., Assimes, T. L., Hemphill, L., and Ma, S. A scoping review of using Large Language Models (LLMs) to investigate Electronic Health Records (EHRs), May 2024a. URL http://arxiv.org/abs/2405. 03066. arXiv:2405.03066 [cs]. Li,...

    work page doi:10.1056/aics2300301

  6. [6]

    Self-alignment pretraining for biomedical entity representations.arXiv preprint arXiv:2010.11784, 2020

    URL http://arxiv. org/abs/2010.11784. arXiv:2010.11784 [cs]. Mansoor, I., Abdullah, M., Rizwan, M. D., and Fraz, M. M. Reasoning with large language models in medicine: a systematic review of techniques, challenges and clinical integration.Health Information Science and Systems, 14 (1):6, November

    work page arXiv 2010

  7. [7]

    doi: 10.1007/ s13755-025-00403-0

    ISSN 2047-2501. doi: 10.1007/ s13755-025-00403-0. URL https://doi.org/10. 1007/s13755-025-00403-0. Navarro, D. F., Magrabi, F., and Coiera, E. Eval- uation format, not model capability, drives triage failure in the assessment of consumer health AI, March

    work page 2047

  8. [8]

    arXiv:2603.11413 [cs]

    URL http://arxiv.org/abs/ 2603.11413. arXiv:2603.11413 [cs]. Shool, S., Adimi, S., Saboori Amleshi, R., Bitaraf, E., Golpira, R., and Tara, M. A systematic review of large language model (LLM) evaluations in clinical medicine.BMC medical informatics and decision mak- ing, 25(1):117, March

    work page arXiv

  9. [9]

    doi: 10.1186/s12911-025-02954-4

    ISSN 1472-6947. doi: 10.1186/s12911-025-02954-4. Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moe- sel, C., Hall, D., Duffett, C., Dube, K., Gallagher, T., and McLachlan, S. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record.Jour- nal of the American Medical Informat...

    work page doi:10.1186/s12911-025-02954-4

  10. [10]

    doi: 10.1093/jamia/ocx079

    ISSN 1527-974X. doi: 10.1093/jamia/ocx079. URL https://doi.org/10. 1093/jamia/ocx079. Wang, Y ., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y ., Mehrabi, S., Sohn, S., and Liu, H. Clinical Information Extraction Applications: A 5 MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinical...

    work page doi:10.1093/jamia/ocx079

  11. [11]

    doi: 10.1016/ j.jbi.2017.11.011

    ISSN 1532-0464. doi: 10.1016/ j.jbi.2017.11.011. URL https://pmc.ncbi.nlm. nih.gov/articles/PMC5771858/. Wu, K., Wu, E., Thapa, R., Wei, K., Zhang, A., Suresh, A., Tao, J. J., Sun, M. W., Lozano, A., and Zou, J. MedCaseReasoning: Evaluating and learn- ing diagnostic reasoning from clinical case reports, May

    work page 2017

  12. [12]

    URL http://arxiv.org/abs/2505. 11733. arXiv:2505.11733 [cs]. Yang, X., Zhao, X., and Shen, Z. EHRStruct: A Com- prehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks, April

    work page arXiv

  13. [13]

    resourceType

    URL http://arxiv.org/abs/ 2511.08206. arXiv:2511.08206 [cs]. 6 MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings A. MedCase-Structured Sample Listing 1 shows the truncated version of a representative structured patient bundle from MedCase-Structured. The full sample is available athttps:/...

    work page arXiv 1975