Pith. sign in

REVIEW 4 major objections 6 minor 29 references

Clinical Trials Ontology Engineering with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLMs can replace manual ontology construction for clinical-trial outcomes at a fraction of the cost and time, with chained GPT-4 approaching human-level concept extraction.

desk verdict Cost and time are measured cleanly; the viability claim isn't supported for medical ontologies because the merging step throws away the relations the paper itself calls crucial. read the letter →

arxiv 2412.14387 v1 pith:XZUN3WQL submitted 2024-12-18 cs.AI

Berkan Çak{i}r This is my paper · ORCID
classification cs.AI
keywords largelanguagemodelsontologyengineeringmergingclinicaltrialsOQuaREpromptchainingGPT-4Llama3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large language models can automate the process of building ontologies from clinical-trial outcome descriptions, and that the automation is viable on both cost and time. It benchmarks GPT-3.5, GPT-4, and two Llama 3 sizes against a human-built golden-standard ontology on 50 diabetes trials, measuring OQuaRE quality alongside dollars and seconds per trial. The headline result is that LLM pipelines cost pennies per trial and take minutes rather than hours, with prompt-chained GPT-4 coming closest to human performance on concept extraction. The author argues this addresses the industrial bottleneck of clinical trials outpacing manual processing, while being explicit that the merged ontology is a categorized concept list because relationships between concepts are dropped.

What carries the argument

The machinery has two parts: an ontology-generation prompt that restricts the LLM to placing biomarkers, endpoint scores, measurement tools, and questionnaires as subclasses of four fixed classes in OWL ontologies, and a merging stage that processes each clinical-trial ontology triple by triple, using a sorted synonym list to decide whether an entity already exists. The sorted list makes lookup O(log n) and the overall merge O(n), at the cost of discarding all object-property relationships. The evaluation engine is the OQuaRE framework, whose NOCOnto metric—measuring concept-extraction density—serves as the main quality barometer because the merged ontology contains no relations to assess.

What would settle it

Query the merged ontology with a relation-dependent medical question—for example, which measurement tools were used to assess which endpoint scores in the diabetes trials—and observe that the merge step has discarded all object-property assertions, so no answer can be recovered; if a practitioner needs that answer, the claimed viability of the resulting artifact fails.

Watch

Extended reading notes

Core claim

The central claim is that a prompt-based LLM pipeline—using reference material, explicit instructions, a reference ontology template, and optional prompt chaining—can extract biomarkers, endpoint scores, measurement tools, and questionnaires from clinical-trial outcome text and merge them into a single ontology at far lower cost and time than a human, with acceptable quality. The strongest evidence is the extrapolation from 50 trials to roughly 6,200 diabetes trials on the public trial registry: chained GPT-4 would cost about $584 and take about 15 days, versus an estimated $31,000 and 1,550 human hours. On the NOCOnto quality metric, larger and chained models extract more concepts, and chained GPT-4 performs best, while open-weight Llama 3 models offer similar cost and time savings. The paper also proposes a linear-time ontology merging method built on a sorted synonym list, with the explicit drawback that all relationships between concepts are lost during merging.

Load-bearing premise

The central claim collapses if a concept list with no relationships between concepts is not a usable ontology for medical practice; the paper itself concedes that relationships are crucial for practical medical applications.

Editorial extensions

If this is right

  • Extrapolating the measured chained-GPT-4 figures from 50 trials to roughly 6,200 diabetes trials implies the full set could be processed for about $584 in about 15 days, versus an estimated $31,000 and 1,550 hours for a human.
  • Prompt chaining raises the fraction of syntactically valid, mergeable ontologies; chained GPT-4 jumps from 26% to 86% of trials included.
  • At per-trial costs of a few cents to a few dollars, real-time ingestion of newly posted trial results becomes economically feasible, so a clinical-trial ontology could be refreshed as trials publish.
  • The merged ontology is a categorized concept list, so its immediate use is limited to finding which concepts belong to which category; it cannot support reasoning over relationships between concepts.
  • Open-weight models such as Llama 3 offer the same order-of-magnitude cost and time savings as API models, which matters for deployments that cannot send clinical data to external services.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the synonym-list merge to canonicalize relation triples as well as entity triples would preserve relationships while keeping the O(n) scaling; the paper identifies relationships as the gap, so this is a natural next step rather than a claim it makes.
  • A direct test of medical utility—for instance, using the merged ontology for a relation-dependent task such as matching patient characteristics to trial eligibility criteria—would show whether the dropped relations actually matter; the paper does not run such a test.
  • Because the NOCOnto metric counts extracted concepts without checking whether they appear in the source trial, an audit that verifies extracted concepts against the trial text would separate true extraction quality from hallucinated concepts.
  • The methodology's restriction to one disease condition suggests a transfer test on multiple conditions: the fixed four-category template may need to be extended or made condition-dependent before multi-condition trials can be integrated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes and evaluates an LLM-based pipeline for building a clinical-trial ontology from ClinicalTrials.gov outcome text. Fifty diabetes trials are processed one by one; each trial's primary and secondary outcomes are passed to GPT-3.5, GPT-4, or Llama-3 (8B/70B) with a one-shot OWL prompt, optionally with prompt chaining; the resulting per-trial ontologies are merged into one ontology by de-duplicating triples through a sorted synonym list. The authors report per-trial cost, time, and inclusion rate, and compare the merged ontologies with a human-built golden standard using the NOCOnto OQuaRE metric. They conclude that LLMs are a viable, far cheaper, and faster replacement for manual ontology construction.

Significance. The cost and time measurements are concrete and the GitHub artifact adds reproducibility; the O(n) synonym-list merging idea is transparent and simple. If the claims are reframed as concept extraction and list construction, the study is a useful feasibility data point. However, the quality evaluation is thin (one metric, no error bars, no relation checking), and the merging step's deletion of all relations means the output is a categorized list rather than a relational ontology. As stated, the central 'viable ontology' claim is not established.

major comments (4)
  1. [§3.4 and §5] The merging method explicitly discards relations, with §3.4 stating that 'the relations between them are lost' and that the ontology 'effectively functions as a categorized list,' while §5 says relationships 'are crucial for practical medical applications.' These two statements jointly contradict the central claim in §1 and §6 that LLMs can automate creation of ontologies usable in the medical field. The measured cost and time reductions apply to constructing a categorized list, not to building the relational ontology the paper motivates. The revision should either preserve relations in the merge or reframe the paper's claim as 'ontology-aligned concept extraction' with a clear explanation of why a categorized list is useful despite lacking relations.
  2. [§4.1, Tables 1 and 3] The human baseline is extrapolated from n=14 trials to per-trial averages, and no variance, confidence intervals, or statistical tests are reported for any cost, time, or quality comparison. Consequently, the §1 claim that GPT-4 'showcases performance near human performance' is unsupported: GPT-4 alone has a 26% inclusion rate, chainedGPT-4 has 86%, and the quality comparison rests on a single metric. The §6 extrapolation from n=50 to 6,200 trials assumes inclusion rates and per-trial costs remain fixed, which is not justified given the observed variability across models and prompting modes.
  3. [§4.2 and §4.4] The OQuaRE evaluation uses only NOCOnto, and §4.4 admits this metric does not capture hallucinated concepts, incorrect categorization, or relations. Since NOCOnto is a concept-count proxy, the Figure 2 ranking of chainedGPT4 as 'best performing' is not evidence of ontology quality. The stated goal of comparing ontology quality is therefore not met; at most, the paper compares concept-extraction rate under a fixed prompt schema.
  4. [§3.3 and §4.2] The prompts include 'the base ontology structure that was used for the golden-standard ontology' (§3.3, prompt tactic 3). Because the LLM outputs are constrained by the same class skeleton used to construct the reference, the NOCOnto comparison partly measures prompt adherence and is biased toward the golden standard. This is not circular reasoning, but it limits the claim that the models are independently 'creating an ontology from clinical trials'; a schema-agnostic evaluation would be needed to support the general ontology-engineering claim.
minor comments (6)
  1. [Abstract] The sentence 'Allowing the medical industry to stay up-to-date with medical developments.' is a sentence fragment and should be integrated into the preceding sentence.
  2. [§3.2] 'inherit randomness' and 'transfer models' should be 'inherent randomness' and 'transformer models'; the spelling 'LLama' is inconsistent throughout (e.g., Table 1 vs. the text).
  3. [§4.2, Figure 2] The note 'GPT4v4 results are unknown and thus not included' is unclear; the paper never defines the 'v4' variant, so the figure's excluded condition is ambiguous.
  4. [Listing 1.3] The pseudo-code appears to invert the deduplication condition described in §3.4: it says 'If entity of interest is present in synonymList: Add triple to mainOntology...,' whereas the text says to add when absent and skip when present. Please correct the listing to match the text.
  5. [Tables 1-3] The table captions state 'n=50 for each model,' but the human rows are extrapolated from n=14; this should be stated in each caption for clarity.
  6. [§4.2] The OQuaRE-NOCOnto definition quoted in §4.2 is confusing as written ('Mean number of the direct superclasses per class minus the subclasses of Thing'); please clarify the exact computation used by the OQuaRE-Metrics tool.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's cost, time, and ontology-quality claims rest on external measurements and an external golden-standard ontology, not on self-referential derivation.

full rationale

The paper's central claims are evaluated by direct measurement against an external human-built golden-standard ontology and by token/GPU-hour cost accounting. No fitted parameters are used, no result is defined in terms of the quantity it is said to predict, and there are no load-bearing self-citations or imported uniqueness theorems. The main methodological caveat is that the prompt includes the base ontology structure used for the golden standard (Section 3.3, tactic 3; Listing 1.1), so NOCOnto scores partly measure prompt compliance; however, this is an evaluation bias, not circular reasoning, because the extracted concept sets are not entailed by the prompt and the cost/time results are independent of quality metrics. Likewise, the paper's own admission that the merging step drops relations (Sections 3.4 and 5) undermines the practical-viability conclusion but does not make the derivation circular. Therefore no step reduces by construction to its own input.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or free parameters. It relies on several domain assumptions about dataset representativeness, the quality metric, the human baseline, and the synonym-based merging correctness.

assumptions (4)
  • domain assumption The 50 diabetes trials from clinicaltrials.gov are representative of all relevant clinical trials.
    The study generalizes from n=50 diabetes-only trials to all 6200 diabetes trials in the conclusion (Section 6).
  • domain assumption The human-generated gold standard ontology is a fair benchmark.
    The gold standard was created by a human layman, not an ontology expert, which may skew the comparison (Section 1, Section 3.3).
  • domain assumption The OQuaRE NOCOnto metric is an adequate measure of ontology quality for this task.
    NOCOnto only counts concept extraction and ignores categorization errors, hallucinations, and relationships (Section 4.2, Section 4.4).
  • domain assumption The synonym list generated by the LLM correctly identifies semantically equivalent concepts.
    The merging algorithm relies on LLM-generated synonyms to deduplicate triples; no validation of this step is reported (Section 3.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clinical Trials Ontology Engineering with Large Language Models." pith.science (2026). https://pith.science/paper/XZUN3WQL

@misc{pith2026241214387,
  author       = {Pith},
  title        = {Pith review of: Clinical Trials Ontology Engineering with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XZUN3WQL}},
  note         = {Machine review of arXiv:2412.14387}
}
read the original abstract

Managing clinical trial information is currently a significant challenge for the medical industry, as traditional methods are both time-consuming and costly. This paper proposes a simple yet effective methodology to extract and integrate clinical trial data in a cost-effective and time-efficient manner. Allowing the medical industry to stay up-to-date with medical developments. Comparing time, cost, and quality of the ontologies created by humans, GPT3.5, GPT4, and Llama3 (8b & 70b). Findings suggest that large language models (LLM) are a viable option to automate this process both from a cost and time perspective. This study underscores significant implications for medical research where real-time data integration from clinical trials could become the norm.

Figures

Figures reproduced from arXiv: 2412.14387 by the authors.

Figure 1
Figure 1. High-overview of proposed methodology. For the study, 50 clinical trials obtained from the clinicaltrials.gov website were used to obtain the results. The CSV file contains the NCT (unique id for tri￾als submitted on clinicaltrial.gov), primary outcomes, and secondary outcomes. The CSV file containing the clinical trials is fed one clinical trial at a time to be processed. For each clinical trial the LLM is asked to… view at source ↗
Figure 2
Figure 2. NOCOnto metric across different LLMs and or used techniques. NO￾COnto(normalised) takes into account the differences in ontologies not being included due to syntax errors (See [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. All OQuaRE metrics across different LLMs and or used techniques. Note that GPT4v4 results are unknown and thus not included. Also note that NOCOnto OQuaRE metric has been separated due to formatting [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 15 canonical work pages

  1. [1]

    — github.com.https://github.com/ollama/ollama, [Accessed 16-06-2024]

    GitHub - ollama/ollama: Get up and running with Llama 3, Mistral, Gemma, and other large language models. — github.com.https://github.com/ollama/ollama, [Accessed 16-06-2024]

    work page 2024

  2. [2]

    https: //arxiv.org/abs/2302.13971, [Accessed 15-06-2024]

    LLaMA: Open and Efficient Foundation Language Models — arxiv.org. https: //arxiv.org/abs/2302.13971, [Accessed 15-06-2024]

    arXiv 2024

  3. [3]

    https://llama.meta.com/llama3/, [Accessed 16-06-2024]

    Meta Llama 3 — llama.meta.com. https://llama.meta.com/llama3/, [Accessed 16-06-2024]

    work page 2024

  4. [4]

    https://doi.org/10.1371/journal.pmed.1000326, https://www.ncbi

    Bastian, H., Glasziou, P., Chalmers, I.: Seventy-Five Trials and Eleven System- atic Reviews a Day: How Will We Ever Keep Up? PLoS Medicine 7(9), e1000326 (Sep 2010). https://doi.org/10.1371/journal.pmed.1000326, https://www.ncbi. nlm.nih.gov/pmc/articles/PMC2943439/

    doi:10.1371/journal.pmed.1000326 2010

  5. [5]

    Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text (2019)

    work page 2019

  6. [6]

    Scientific American 284(5), 34–43 (2001), https://www.jstor.org/stable/26059207, publisher: Sci- entific American, a division of Nature America, Inc

    Berners-Lee, T., Hendler, J., Lassila, O.: The Semantic Web. Scientific American 284(5), 34–43 (2001), https://www.jstor.org/stable/26059207, publisher: Sci- entific American, a division of Nature America, Inc

    work page arXiv 2001

  7. [7]

    https://doi.org/10.48550/arXiv.2005.14165, http://arxiv.org/abs/2005.14165, arXiv:2005.14165 [cs]

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Win- ter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford...

    arXiv doi:10.48550/arxiv.2005.14165 2020

  8. [8]

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M.T., Zhang, Y.: Sparks of artificial general intelligence: Early experiments with gpt-4 (2023)

    work page 2023

Show all 29 references
  1. [9]

    https://doi.org/10.48550/arXiv.1805.03885, http://arxiv.org/abs/1805.03885, arXiv:1805.03885 [cs]

    Chah, N.: OK Google, What Is Your Ontology? Or: Exploring Free- base Classification to Understand Google’s Knowledge Graph (May 2018). https://doi.org/10.48550/arXiv.1805.03885, http://arxiv.org/abs/1805.03885, arXiv:1805.03885 [cs]

    2018 doi arXiv

  2. [10]

    In: Proceedings of the 3rd Clinical Natural Language Pro- cessing Workshop

    Chen, M., Lan, G., Du, F., Lobanov, V.: Joint Learning with Pre-trained Transformer on Named Entity Recognition and Relation Extraction Tasks for Clinical Analytics. In: Proceedings of the 3rd Clinical Natural Language Pro- cessing Workshop. pp. 234–242. Association for Comput...

    2020 doi

  3. [11]

    org/abs/1810.04805v2

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding (Oct 2018), https://arxiv. org/abs/1810.04805v2

    2018 arXiv

  4. [12]

    Expert Systems with Applica- tions 40(7), 2696–2703 (Jun 2013)

    Duque-Ramos, A., Fern´ andez-Breis, J.T., Iniesta, M., Dumontier, M., Ega˜ na Aranguren, M., Schulz, S., Aussenac-Gilles, N., Stevens, R.: Evaluation of the oquare framework for ontology quality. Expert Systems with Applica- tions 40(7), 2696–2703 (Jun 2013). https://doi.org/1...

    2013 doi

  5. [13]

    Giglou, H.B., D’Souza, J., Auer, S.: Llms4ol: Large language models for ontology learning (2023)

    2023

  6. [14]

    https://doi.org/10.48550/arXiv.1912.13415, http://arxiv

    Giorgi, J., Wang, X., Sahar, N., Shin, W.Y., Bader, G.D., Wang, B.: End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Clinical Trials Ontology Engineering with Large Language Models 13 Models (Dec 2019). https://doi.org/10.48550/arXiv.1912....

    2019 doi arXiv

  7. [15]

    (eds.) Handbook on Ontologies, pp

    Guarino, N., Oberle, D., Staab, S.: What Is an Ontology? In: Staab, S., Studer, R. (eds.) Handbook on Ontologies, pp. 1–17. International Handbooks on Information Systems, Springer, Berlin, Heidelberg (2009). https://doi.org/10.1007/978-3-540- 92673-3 0

    2009 doi

  8. [16]

    https://techcommunity.microsoft.com/t5/user/viewprofilepage/user id/2208109: Comparing GPT-3.5 & GPT-4: A Thought Framework on When To Use Each Model — techcommunity.microsoft.com, [Accessed 17-06-2024]

    2024

  9. [17]

    ACM Comput- ing Surveys 55(12), 248:1–248:38 (Mar 2023)

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of Hallucination in Natural Language Generation. ACM Comput- ing Surveys 55(12), 248:1–248:38 (Mar 2023). https://doi.org/10.1145/3571730, https://dl.acm.org/doi/10.1145/3571730

    2023 doi

  10. [18]

    BMC Medical Informatics and Decision Making 10(1), 56 (Sep 2010)

    Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., Sim, I.: Ex- aCT: automatic extraction of clinical trial characteristics from journal publications. BMC Medical Informatics and Decision Making 10(1), 56 (Sep 2010). https://doi.org/10.1186/1472-6947-10-56, https://doi.or...

    2010 doi

  11. [19]

    https://doi.org/10.48550/arXiv.2302.08091, http://arxiv.org/abs/2302.08091, arXiv:2302.08091 [cs]

    Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M.J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., Alsentzer, E.: Do We Still Need Clin- ical Language Models? (Feb 2023). https://doi.org/10.48550/arXiv.2302.08091, http://arxiv.org/abs/2302.08091, arXiv:2302.0...

    2023 doi arXiv

  12. [20]

    In: 2021 IEEE 9th International Con- ference on Healthcare Informatics (ICHI)

    Liu, X., Hersch, G.L., Khalil, I., Devarakonda, M.: Clinical Trial Infor- mation Extraction with BERT. In: 2021 IEEE 9th International Con- ference on Healthcare Informatics (ICHI). pp. 505–506 (Aug 2021). https://doi.org/10.1109/ICHI52183.2021.00092, iSSN: 2575-2634

    2021

  13. [21]

    Matentzoglu, N., Caufield, J.H., Hegde, H.B., Reese, J.T., Moxon, S., Kim, H., Harris, N.L., Haendel, M.A., Mungall, C.J.: Mappergpt: Large language models for linking and mapping entities (2023)

    2023

  14. [22]

    AMIA Summits on Translational Science Proceedings 2021, 485–494 (May 2021), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8378650/

    Nye, B.E., DeYoung, J., Lehman, E., Nenkova, A., Marshall, I.J., Wallace, B.C.: Understanding Clinical Trial Reports: Extracting Medical Entities and Their Rela- tions. AMIA Summits on Translational Science Proceedings 2021, 485–494 (May 2021), https://www.ncbi.nlm.nih.gov/pmc...

    2021

  15. [23]

    OpenAI: Api reference (2023), https://platform.openai.com/docs/ api-reference/audio/createTranscription

    2023

  16. [24]

    OpenAI: Prompt engineering (2023), https://platform.openai.com/docs/ guides/prompt-engineering

    2023

  17. [25]

    International Journal of Computing p

    Palagin, O., Kaverinskiy, V., Litvin, A., Malakhov, K.: Ontochatgpt in- formation system: Ontology-driven structured prompts for chatgpt meta- learning. International Journal of Computing p. 170–183 (Jul 2023). https://doi.org/10.47839/ijc.22.2.3086, http://dx.doi.org/10.47839...

    2023 doi

  18. [26]

    Engineering Applications of Artificial Intelligence 63, 108–125 (Aug 2017)

    Rani, M., Dhar, A.K., Vyas, O.P.: Semi-automatic terminology ontology learn- ing based on topic modeling. Engineering Applications of Artificial Intelligence 63, 108–125 (Aug 2017). https://doi.org/10.1016/j.engappai.2017.05.006, https: //www.sciencedirect.com/science/article/...

    2017 doi

  19. [27]

    https://doi.org/10.48550/arXiv.2005.12833, http: //arxiv.org/abs/2005.12833, arXiv:2005.12833 [cs] 14 C ¸ akır et al

    Rasmy, L., Xiang, Y., Xie, Z., Tao, C., Zhi, D.: Med-BERT: pre-trained con- textualized embeddings on large-scale structured electronic health records for disease prediction (May 2020). https://doi.org/10.48550/arXiv.2005.12833, http: //arxiv.org/abs/2005.12833, arXiv:2005.128...

    2020 doi arXiv

  20. [28]

    tecnomod um, U.o.M.: oquare-metrics (2023), https://github.com/tecnomod-um/ oquare-metrics

    2023

  21. [29]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need (Jun 2017),https://arxiv.org/abs/ 1706.03762v5

    2017 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.