Pith. sign in

REVIEW 1 major objections 7 minor 22 references

Converting Annotated Clinical Cases into Structured Case Report Forms

T0 review · 1 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper claims a general semi-automatic methodology that converts annotated clinical case corpora into gold-standard-filled Case Report Forms, with zero-shot GPT-4o reaching 59.7% micro F1 in Italian and 67.3% in English on the…

desk verdict Useful new CRF resource from E3C, but test item construction leaks test annotations into the benchmark and the reported F1 needs a fix before it is a clean result. read the letter →

arxiv 2506.11666 v1 pith:Q62HNRZV submitted 2025-06-13 cs.CL cs.AI

classification cs.CLcs.AI
keywords CaseReportFormsCRFslotfillingclinicalnoteszero-shotLLMevaluationE3CcorpusdatasetconversionmultilingualNLPinformationextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Case Report Forms (CRFs) are standard instruments for collecting patient data in clinical research, but publicly available, well-annotated CRF datasets are scarce. This paper proposes a semi-automatic methodology that repurposes clinical-case corpora already annotated for information extraction into structured, gold-standard-filled CRFs, and applies it to the E3C corpus in Italian and English. The released dataset defines a CRF slot-filling task with three sub-tasks: diagnosis, clinical history, and exam-result extraction. In zero-shot evaluation, GPT-4o reaches 59.7% micro F1 on Italian and 67.3% on English, with open-weight models behind, showing that CRF filling remains difficult even for recent large language models.

What carries the argument

The load-bearing mechanism is a two-step conversion pipeline. First, clinical cases are clustered using a graph whose edge weight is $s=3d+e$ (for E3C, $s=3d+\frac{1}{2}(e+b)$), where $d$ is diagnosis similarity computed from UMLS-augmented embeddings and $e,b$ are ratios of shared clinical entities and body parts; Louvain community detection over weakly connected components forms the groups. Second, each group receives a CRF generated by collecting annotation-derived items per section, normalizing them via UMLS and manual review, and filling each case from its annotations. The E3C adaptation creates diagnosis, history, and exam items from the corpus's entity, event, relation, and attribute annotations.

What would settle it

Regenerate the E3C CRF item sets using only training-split annotations, rerun the same zero-shot GPT-4o prompts on the test cases, and compare micro F1 to the reported 59.7% (Italian) and 67.3% (English); a substantial drop would show the current scores are inflated by item-set leakage.

Watch

Extended reading notes

Core claim

The central claim is that a clinically meaningful CRF benchmark can be produced from an existing annotated corpus without designing forms from scratch. The pipeline clusters clinical notes by diagnosis-weighted semantic similarity, creates one group-specific CRF item set per cluster, and fills each case's CRF from its annotations to produce gold labels. Applied to E3C, this yields three sub-tasks on which zero-shot GPT-4o is the strongest tested model, reaching 59.7% micro F1 (Italian) and 67.3% (English); exam-result generation is the hardest sub-task, and models score about 7.5 micro-F1 points higher on English than on Italian on average.

Load-bearing premise

The load-bearing premise is that generating the group-specific CRF item sets from the whole corpus, including test cases, does not bias the evaluation; if that assumption fails, the reported F1 scores overstate how well the models can fill CRFs for unseen notes.

Editorial extensions

If this is right

  • Other annotated clinical corpora can be converted into CRF benchmarks with the same pipeline, lowering the cost of obtaining task-specific forms without manual design.
  • The released dataset gives a multilingual public testbed for CRF slot filling, allowing future models to be compared on identical items and gold fillings.
  • The strongest models still lag on exam-result extraction, so the benchmark isolates a generative extraction skill rather than only classification.
  • The simplified three-value history schema keeps the dataset usable across languages, while the underlying E3C annotations still allow the full twelve-value schema to be restored.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because group-specific CRFs are built from all cases including test cases, the test items are partly determined by test annotations; a train-only CRF version would likely yield lower scores and would test item generalization more strictly.
  • With about 90% of CRF items left unfilled, micro F1 is dominated by correct 'not available' answers, so the task-level precision and recall the paper reports are the more informative comparison.
  • A natural next experiment is to fine-tune an open-weight model on the training split and measure the gap to zero-shot GPT-4o, since all reported numbers are from zero-shot prompting.
  • The methodology's portability to other datasets depends on matching the available annotation types to CRF sections; the paper itself notes that treatment items were excluded because E3C lacks treatment annotations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper proposes a semi-automatic methodology for converting information-extraction-annotated clinical case corpora into structured, filled Case Report Forms (CRFs). The method clusters clinical notes by semantic similarity (with diagnosis as the primary dimension), generates a group-specific CRF item set per cluster, and populates it from the original annotations. The methodology is applied to the E3C corpus in English and Italian, and the resulting dataset is released publicly. The authors evaluate several LLMs in a zero-shot setting, reporting that GPT-4o achieves 59.7% micro F1 on Italian and 67.3% on English, with open-weight models performing worse, and conclude that CRF slot filling remains challenging.

Significance. If the evaluation is valid, the released dataset constitutes a new public benchmark for CRF filling, a task for which annotated datasets are scarce. The methodology itself is a useful contribution and is described with enough detail to be adapted to other IE corpora. The paper is transparent about many construction choices and releases the data on Hugging Face, which supports reproducibility. However, the headline F1 scores rest on an evaluation protocol whose validity is questionable (see major comments), so the benchmark's value as a reliable measurement tool is not yet established.

major comments (1)
  1. [§3.1 and §5.1 (diagnosis extraction, item normalization)] GPT-4o is used both during dataset construction (for diagnosis selection in §3.1 and for the 'close source Large Language Model' that proposes item mappings during data revision in §3.2) and as the best-performing evaluation model. This creates a potential circularity: the gold answers and item phrasing are shaped by the same model family that is later evaluated against them, which could inflate GPT-4o's scores relative to open-weight models that had no role in dataset construction. The manual review of 10 examples is a positive step but does not rule out systematic favoritism. The authors should discuss this risk and, if feasible, report an additional evaluation using a model not involved in dataset construction, or at least quantify the overlap between GPT-4o's construction outputs and its test predictions.
minor comments (7)
  1. [Abstract] The text reads 'We release the datest at ...' — 'datest' should be 'dataset'.
  2. [§3.1 vs. §5.1] The similarity formula is given as s = 3d + e in §3.1 (Eq. 1) but as s = 3d + 1/2(e + b) in §5.1. The relationship between the general formulation and the E3C-specific adaptation should be clarified, including the role of the body-part ratio b.
  3. [Table 3] The 'Filled' column percentages are not clearly tied to the item counts; adding a note on how 'filled' is defined (e.g., value different from 'not available') would improve readability.
  4. [Table 8] Some entries have inconsistent spacing (e.g., '13/ 9.7' and '11 / 9.7'), and the table would benefit from a footnote explaining the units for mean and standard deviation.
  5. [§4 and §6] The phrase 'closed Large Language Models (zero-shot)' is awkward; 'closed' likely means proprietary, and the sentence could be rephrased for clarity.
  6. [Appendix A.2 Table 6] The CRF value for 'Finite' permanence is listed as 'Certainly not chronic', which is confusing because a finite condition is not necessarily 'certainly not chronic'; consider rewording or adding an example.
  7. [§7 (Results)] No confidence intervals or significance tests are reported for the differences between models. Given the small test sizes, a statement about variability would help readers assess the robustness of the ranking.

Circularity Check

1 steps flagged · score 5.0 of 10

Test CRF items are generated from all cases (including test cases), so exam and diagnosis questions are defined from the same annotations that provide the gold answers.

  1. self definitional [Section 5.1, 'Train-test split', 'Exams', and 'Diagnosis']
    "group-specific CRFs are generated on all the cases in the corpus. ... constructing comprehensive item sets from the full dataset is necessary and does not introduce bias beyond the task’s inherent structure. ... creating clinical history item sets ... excluding from the final set the ones filled only for test cases after data revision. ... A CRF exam item was created for each textual span with a corresponding RML, representing its filling value. ... For each diagnosis, an item was created and populated with 'yes' if it applied to the clinical case and 'not available' otherwise."

    The test CRF item sets are built from annotations of all cases, including test cases. Exam items are one-to-one with textual spans carrying an RML (the value the model must predict), and diagnosis items are one-to-one with diagnoses selected from the case annotations. An annotation appearing only in a test case therefore generates a test item whose gold value is exactly that annotation. The authors exclude test-only items for clinical history but do not describe the same exclusion for exams or diagnosis; the history filter shows they regard test-only items as a contamination source.

full rationale

The claimed headline results (GPT-4o micro-F1 59.7 on Italian, 67.3 on English) are computed on test items whose sets are generated from the full corpus, including the test cases themselves. For exams, every item is created from a textual span linked to an RML, i.e., from the annotation that also supplies the gold value; for diagnosis, every item comes from a diagnosis selected from the case annotations. The authors explicitly filter out test-only items for clinical history but not for exams or diagnosis, so the test questions are partly selected from the gold answers. This is a genuine reduction by construction, but it does not make the model outputs tautological: the model still has to read the clinical note and decide on a value, and the paper's claim that CRF filling is challenging retains independent content. The train-test split comes from Ghosh et al. (2025), whose author list overlaps with this paper, but using an existing split is ordinary practice and is not load-bearing evidence. GPT-4o was also used in diagnosis selection and item mapping, but labels were manually reviewed, and the model is evaluated zero-shot, so that overlap is not itself a fitted-input circularity. Overall score 5: one central benchmark-construction step is definitionally coupled to the gold labels, but the quantitative conclusion is not forced.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or abstract entities are invented. The main borrowed assumptions are the reliability of E3C annotations and the clinical relevance of the clustering. The free parameters are hand-tuned similarity weights and an augmentation cap, all of which materially shape the generated CRFs and thus the benchmark.

free parameters (3)
  • Similarity weight for diagnosis (d) = 3
    In s = 3d + 1/2(e+b), the coefficient 3 is hand-chosen to give diagnosis more weight in clustering. This affects group composition and therefore all CRF item sets.
  • Similarity weight for entities and body parts (e and b) = 0.5
    The 1/2 coefficient for shared entity and body part ratios is a manual design choice, not derived from data or theory.
  • Maximum UMLS related terms per mention = 5
    Augmenting each term with up to 5 UMLS-related concepts is a hand-set limit that influences entity sharing and similarity scores.
assumptions (4)
  • domain assumption E3C annotations are accurate and sufficient to define gold-standard CRF values.
    Section 5.1 builds all items and fill values from E3C spans and attributes without external validation against real CRF experts, aside from a manual review of 10 diagnosis examples.
  • domain assumption The seven CRF areas observed in 200 hospital CRFs generalize to the E3C clinical cases.
    Section 3 states the CRF design is 'informed by an analysis of 200 pairs of clinical notes and populated CRFs from an Italian hospital' and then applies this structure to E3C, a different corpus.
  • domain assumption Clustering based on diagnosis similarity produces clinically coherent CRF item sets.
    Section 3.1 prioritizes diagnosis as the key clustering dimension, but there is no evaluation showing these clusters correspond to meaningful clinical groupings.
  • domain assumption GPT-4o diagnosis extraction is accurate enough for group construction.
    Section 5.1 uses GPT-4o in 4-shot settings to select diagnoses and manually reviews only 10 examples, so the quality of the extracted diagnosis labels for all other notes is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Converting Annotated Clinical Cases into Structured Case Report Forms." pith.science (2026). https://pith.science/paper/Q62HNRZV

@misc{pith2026250611666,
  author       = {Pith},
  title        = {Pith review of: Converting Annotated Clinical Cases into Structured Case Report Forms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q62HNRZV}},
  note         = {Machine review of arXiv:2506.11666}
}
read the original abstract

Case Report Forms (CRFs) are largely used in medical research as they ensure accuracy, reliability, and validity of results in clinical studies. However, publicly available, wellannotated CRF datasets are scarce, limiting the development of CRF slot filling systems able to fill in a CRF from clinical notes. To mitigate the scarcity of CRF datasets, we propose to take advantage of available datasets annotated for information extraction tasks and to convert them into structured CRFs. We present a semi-automatic conversion methodology, which has been applied to the E3C dataset in two languages (English and Italian), resulting in a new, high-quality dataset for CRF slot filling. Through several experiments on the created dataset, we report that slot filling achieves 59.7% for Italian and 67.3% for English on a closed Large Language Models (zero-shot) and worse performances on three families of open-source models, showing that filling CRFs is challenging even for recent state-of-the-art LLMs. We release the datest at https://huggingface.co/collections/NLP-FBK/e3c-to-crf-67b9844065460cbe42f80166

Figures

Figures reproduced from arXiv: 2506.11666 by the authors.

Figure 1
Figure 1. Example of a Case Report Form filled with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Summary of our two-step CRF generation and filling pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Graph representation of the E3C English dataset. Each node is a clinical note and the edges represent the similarity between cases. Darker edges represent higher similarity. The color of the nodes rep￾resents the group assigned by the clustering algorithm. number of corresponding CRF sections. Each task requires filling CRF items based on information extracted from clinical cases, but they may vary in complexity. Fo… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example of a generated CRF for English group 1 and filled with the annotation from the clinical case [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Shantala Bellary, Binny Krishnankutty, and M. S. Latha. 2014. https://doi.org/10.4103/2229-3485.140555 Basics of case report form designing in clinical research . Perspectives in Clinical Research, 5(4):159--166

  4. [4]

    off-the-shelf

    Emma Chiaramello, Francesco Pinciroli, Alberico Bonalumi, Angelo Caroli, and Gabriella Tognola. 2016. https://doi.org/10.1016/j.jbi.2016.07.017 Use of “off-the-shelf” information extraction algorithms in clinical informatics: A feasibility study of MetaMap annotation of Italian medical notes . Journal of Biomedical Informatics, 63:22--32. @inproceedings g...

  5. [5]

    Aaron Grattafiori et al. 2024. https://arxiv.org/abs/2407.21783 The Llama 3 herd of models . Preprint, arXiv:2407.21783

  6. [6]

    Robert Fleischmann, Anne-Marie Decker, Antje Kraft, Knut Mai, and Sein Schmidt. 2017. https://doi.org/10.1186/s12874-017-0429-y Mobile electronic versus paper case report forms in clinical trials: a randomized controlled trial . BMC Medical Research Methodology, 17(1):153

  7. [7]

    Soumitra Ghosh, Begona Altuna, Saeed Farzi, Pietro Ferrazzi, Alberto Lavelli, Giulia Mezzanotte, Manuela Speranza, and Bernardo Magnini. 2025. Low-resource information extraction with the European Clinical Case Corpus . ArXiv preprint

  8. [8]

    Dyer, Faridis Serrano, Carmen Watrin, Pierre Pages, Sajad Mousavi, Anil Degala, Jessica Lyons, Danielle Pillion, Joany M

    Alba Gutiérrez-Sacristán, Simran Makwana, Audrey Dionne, Simran Mahanta, Karla J. Dyer, Faridis Serrano, Carmen Watrin, Pierre Pages, Sajad Mousavi, Anil Degala, Jessica Lyons, Danielle Pillion, Joany M. Zachariasse, Lara S. Shekerdemian, Dongngan T. Truong, Jane W. Newburger, and Paul Avillach. 2024. https://doi.org/10.1016/j.ebiom.2024.105337 Developmen...

Show all 22 references
  1. [9]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the ACM SIGOPS 29th Symposium on Operating ...

  2. [10]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. https://arxiv.org/abs/2405.17428 NV-Embed : Improved techniques for training LLMs as generalist embedding models . Preprint, arXiv:2405.17428

  3. [11]

    Ching-Heng Lin, Nai-Yuan Wu, and Der-Ming Liou. 2015. https://doi.org/10.1016/j.jbi.2014.08.013 A multi-technique approach to bridge electronic case report form design and data standard adoption . Journal of Biomedical Informatics, 53:49--57

  4. [12]

    Hao Lu, Mahantesh Halappanavar, and Ananth Kalyanaraman. 2015. https://doi.org/10.1016/j.parco.2015.03.003 Parallel heuristics for scalable community detection . Parallel Computing, 47:19--37

  5. [13]

    W. R. Mac Kenzie, A. J. Davidson, A. Wiesenthal, J. P. Engel, K. Turner, L. Conn, S. J. Becker, S. Moffatt, S. L. Groseclose, J. Jellison, J. Stinn, N. Y. Garrett, L. Helmus, B. Harmon, C. L. Richards, J. R. Lumpkin, and M. F. Iademarco. 2016. https://doi.org/10.1177/003335491...

  6. [14]

    Bernardo Magnini, Bego \ n a Altuna, Alberto Lavelli, Anne-Lyse Minard, Manuela Speranza, and Roberto Zanoli. 2023. https://doi.org/10.1007/978-3-031-17258-8_17 European Clinical Case Corpus , pages 283--288. Springer International Publishing, Cham

  7. [15]

    National Library of Medicine (US) . 2024. http://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html UMLS knowledge sources [dataset on the internet]. release 2024aa . Cited 2025 March

  8. [16]

    OpenAI and Josh Achiam et al. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

  9. [17]

    Frank Pétavy et al. 2019. https://www.appliedclinicaltrialsonline.com/view/global-standardization-clinical-research-data Global standardization of clinical research data . Applied Clinical Trials, 28(4):20--23

  10. [18]

    Qwen and An Yang et al. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

  11. [19]

    Richesson and Prakash Nadkarni

    Rachel L. Richesson and Prakash Nadkarni. 2011. https://doi.org/10.1136/amiajnl-2011-000107 Data standards for clinical research data collection forms: current status and challenges . Journal of the American Medical Informatics Association, 18(3):341--346

  12. [20]

    Eugenia Rinaldi, Caroline Stellmach, and Sylvia Thun. 2025. https://doi.org/10.2196/51598 How to design electronic case report form (ecrf) questions to maximize semantic interoperability in clinical research . Interactive Journal of Medical Research, 14:e51598

  13. [21]

    Dongkuan Xu and Yingjie Tian. 2015. https://doi.org/10.1007/s40745-015-0040-1 A comprehensive survey of clustering algorithms . Annals of Data Science, 2(2):165--193

  14. [22]

    Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. https://arxiv.org/abs/2412.19048 Jasper and Stella : distillation of SOTA embedding models . Preprint, arXiv:2412.19048

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.