{"id":"0782552b-91c6-404e-8822-9c17a839997e","arxiv_id":"2412.14387","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs can generate and merge clinical trial ontologies at about one-thousandth of human cost, but with lower quality and no relationships between concepts.","lead":"This paper tests whether ChatGPT and Llama models can automatically turn clinical trial results into structured medical ontologies, comparing cost, speed, and quality against human-made ontologies. The study finds LLMs are much cheaper and faster, but the resulting ontologies are simple category lists and often fail syntax checks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The merging step discards all relationships, and the paper concedes relationships are crucial for medical use; the cost/time results demonstrate efficient list construction, not viable ontology creation.","rationale":"The reader's weakest-assumption identification is the same load-bearing concern I find. The strongest claim is about viability of LLM-based ontology creation in a medical context; the paper's own methodology removes all relations, and the paper explicitly labels relationship omission as a significant limitation with relationships crucial for medical applications. This is an internally sourced admission, not an external objection. The cost and time measurements are clear, reproducible code is provided, and GPT temperature was set to 0; these support the practical efficiency of concept-list construction. However, the evaluation metric (NOCOnto) is structurally unable to detect the missing relations, so the quality component of the viability claim is unsupported. The paper is transparent about the limitation, and the cost/time contribution stands independently, so I do not recommend REJECT. Instead, the conclusion overreaches by calling the output an ontology usable by those in the medical field. My verdict remains CONDITIONAL, matching the reader's verdict, because the central claim survives only if a categorized list is accepted as sufficient for the medical use case, which the paper itself disputes. The proposed test—preserving object properties and measuring relation quality—would settle whether this concern actually lands.","tokens_in":8854,"tokens_out":3544,"duration_ms":30049,"concrete_test":"Re-run the pipeline on the same 50 clinical trials with a merging step that preserves OWL object property axioms (e.g., hasBiomarker, measures, endpointOf) instead of dropping non-subclass triples. Then compute relation-level precision and recall against a physician-curated gold standard for the same trials. If the LLM-generated ontologies cannot capture gold-standard relations at acceptable F1, or if clinicians cannot use the relation-free lists to answer basic queries such as 'which measurement tools assess HbA1c?', the viability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLMs are a viable option to automate the creation of clinical-trial ontologies for medical practice (§1, §6). This claim requires the merged artifact to be a functional ontology. Section 3.4 states that during merging 'the relations between them are lost,' producing an ontology that 'effectively functions as a categorized list' and limiting practitioners to checking related concepts within categories. Section 5 then identifies relationship omission as 'a significant limitation' and states that relationships 'are crucial for practical medical applications.' These statements come from the paper itself and jointly undermine the strongest claim: the produced artifact is not a relational ontology and therefore does not match the medical use case the paper motivates. The measured cost and time advantages apply to extracting and de-duplicating concept labels, not to constructing a usable medical ontology. The evaluation (§4.2) uses only the NOCOnto metric, which counts concepts and cannot detect absent or incorrect relations, so no evidence is offered that the relation-free output retains clinical utility. The viability claim holds only if an 'ontology' is redefined as a categorized list, which contradicts the paper's own framing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes and evaluates an LLM-based pipeline for building a clinical-trial ontology from ClinicalTrials.gov outcome text. Fifty diabetes trials are processed one by one; each trial's primary and secondary outcomes are passed to GPT-3.5, GPT-4, or Llama-3 (8B/70B) with a one-shot OWL prompt, optionally with prompt chaining; the resulting per-trial ontologies are merged into one ontology by de-duplicating triples through a sorted synonym list. The authors report per-trial cost, time, and inclusion rate, and compare the merged ontologies with a human-built golden standard using the NOCOnto OQuaRE metric. They conclude that LLMs are a viable, far cheaper, and faster replacement for manual ontology construction.","tokens_in":9128,"tokens_out":5315,"duration_ms":41328,"significance":"The cost and time measurements are concrete and the GitHub artifact adds reproducibility; the O(n) synonym-list merging idea is transparent and simple. If the claims are reframed as concept extraction and list construction, the study is a useful feasibility data point. However, the quality evaluation is thin (one metric, no error bars, no relation checking), and the merging step's deletion of all relations means the output is a categorized list rather than a relational ontology. As stated, the central 'viable ontology' claim is not established.","major_comments":[{"comment":"The merging method explicitly discards relations, with §3.4 stating that 'the relations between them are lost' and that the ontology 'effectively functions as a categorized list,' while §5 says relationships 'are crucial for practical medical applications.' These two statements jointly contradict the central claim in §1 and §6 that LLMs can automate creation of ontologies usable in the medical field. The measured cost and time reductions apply to constructing a categorized list, not to building the relational ontology the paper motivates. The revision should either preserve relations in the merge or reframe the paper's claim as 'ontology-aligned concept extraction' with a clear explanation of why a categorized list is useful despite lacking relations.","section":"§3.4 and §5"},{"comment":"The human baseline is extrapolated from n=14 trials to per-trial averages, and no variance, confidence intervals, or statistical tests are reported for any cost, time, or quality comparison. Consequently, the §1 claim that GPT-4 'showcases performance near human performance' is unsupported: GPT-4 alone has a 26% inclusion rate, chainedGPT-4 has 86%, and the quality comparison rests on a single metric. The §6 extrapolation from n=50 to 6,200 trials assumes inclusion rates and per-trial costs remain fixed, which is not justified given the observed variability across models and prompting modes.","section":"§4.1, Tables 1 and 3"},{"comment":"The OQuaRE evaluation uses only NOCOnto, and §4.4 admits this metric does not capture hallucinated concepts, incorrect categorization, or relations. Since NOCOnto is a concept-count proxy, the Figure 2 ranking of chainedGPT4 as 'best performing' is not evidence of ontology quality. The stated goal of comparing ontology quality is therefore not met; at most, the paper compares concept-extraction rate under a fixed prompt schema.","section":"§4.2 and §4.4"},{"comment":"The prompts include 'the base ontology structure that was used for the golden-standard ontology' (§3.3, prompt tactic 3). Because the LLM outputs are constrained by the same class skeleton used to construct the reference, the NOCOnto comparison partly measures prompt adherence and is biased toward the golden standard. This is not circular reasoning, but it limits the claim that the models are independently 'creating an ontology from clinical trials'; a schema-agnostic evaluation would be needed to support the general ontology-engineering claim.","section":"§3.3 and §4.2"}],"minor_comments":[{"comment":"The sentence 'Allowing the medical industry to stay up-to-date with medical developments.' is a sentence fragment and should be integrated into the preceding sentence.","section":"Abstract"},{"comment":"'inherit randomness' and 'transfer models' should be 'inherent randomness' and 'transformer models'; the spelling 'LLama' is inconsistent throughout (e.g., Table 1 vs. the text).","section":"§3.2"},{"comment":"The note 'GPT4v4 results are unknown and thus not included' is unclear; the paper never defines the 'v4' variant, so the figure's excluded condition is ambiguous.","section":"§4.2, Figure 2"},{"comment":"The pseudo-code appears to invert the deduplication condition described in §3.4: it says 'If entity of interest is present in synonymList: Add triple to mainOntology...,' whereas the text says to add when absent and skip when present. Please correct the listing to match the text.","section":"Listing 1.3"},{"comment":"The table captions state 'n=50 for each model,' but the human rows are extrapolated from n=14; this should be stated in each caption for clarity.","section":"Tables 1-3"},{"comment":"The OQuaRE-NOCOnto definition quoted in §4.2 is confusing as written ('Mean number of the direct superclasses per class minus the subclasses of Thing'); please clarify the exact computation used by the OQuaRE-Metrics tool.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The cost and time data are useful and reproducible, but the paper's central claim is substantially overstated relative to what the evaluation can support. The authors may be able to reframe the contribution as ontology-aligned concept extraction and add a proper relation-preserving merge or a validation on real clinical queries. I would not reject the manuscript outright; I would require a major revision that either preserves relations or recalibrates the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Cost and time are measured cleanly; the viability claim isn't supported for medical ontologies because the merging step throws away the relations the paper itself calls crucial. Worth a look as an empirical baseline, not as a solution.\n\nThe paper compares GPT-3.5, GPT-4, and Llama-3 8b/70b, with and without prompt chaining, on extracting ontology skeletons from 50 diabetes trial records. The number that stands out is cost per trial: about $0.006 for GPT-3.5 and $0.06 for GPT-4, versus a human estimate of $5; time per trial drops from minutes to seconds. The GitHub repo with code, prompts, and data is real, and that is the strongest part of the package. The O(n) merge via a sorted synonym list is simple but legitimate, and the paper is honest about swapping relational structure for scalability.\n\nThe evaluation is the weak link. It uses a single OQuaRE metric, NOCOnto, which counts children and cannot see hallucinated or misplaced concepts. The human baseline is extrapolated from 14 trials with a fixed $20/hour wage, with no variance. Inclusion rates range from 24% to 86% and the paper suggests retrying with different seeds in production, which changes the cost and time estimates beyond what Table 1 shows. Most important, the merge drops all relations, and Section 5 says those relations are 'crucial for practical medical applications.' So the merged artifact is a categorized concept list, not a functional ontology. The stress-test note is correct: the cost/time results demonstrate efficient list construction, not viable ontology creation. This is not a small caveat; it breaks the main claim as worded in the abstract and conclusion.\n\nThe paper is still useful. Anyone building LLM-based extraction pipelines for clinical text will find the per-model cost and inclusion data a cheap reference point. It also works as a teaching example of why a single structural metric is not enough. The author clearly knows the limitations—they are written down—but the title and conclusion overshoot.\n\nMy verdict: this deserves a serious referee for a specialty workshop or applied journal, but only with major revision. The author should either preserve relations (or match against them) or reframe the contribution as concept harvesting from clinical trials, and add per-run variance. If the claim is narrowed, the paper's data stands. As is, I'd read it as a useful negative result on the gap between 'ontology' and 'categorized list.'","headline":"Cost and time are measured cleanly; the viability claim isn't supported for medical ontologies because the merging step throws away the relations the paper itself calls crucial.","tokens_in":9511,"tokens_out":2232,"would_cite":false,"duration_ms":21036,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLMs can replace manual ontology construction for clinical-trial outcomes at a fraction of the cost and time, with chained GPT-4 approaching human-level concept extraction.","keywords":["large language models","ontology engineering","ontology merging","clinical trials","OQuaRE","prompt chaining","GPT-4","Llama 3"],"falsifier":"Query the merged ontology with a relation-dependent medical question—for example, which measurement tools were used to assess which endpoint scores in the diabetes trials—and observe that the merge step has discarded all object-property assertions, so no answer can be recovered; if a practitioner needs that answer, the claimed viability of the resulting artifact fails.","tokens_in":8628,"feed_emoji":"🧬","tokens_out":7971,"duration_ms":63782,"temperature":0.7,"pith_summary":"The paper claims that large language models can automate the process of building ontologies from clinical-trial outcome descriptions, and that the automation is viable on both cost and time. It benchmarks GPT-3.5, GPT-4, and two Llama 3 sizes against a human-built golden-standard ontology on 50 diabetes trials, measuring OQuaRE quality alongside dollars and seconds per trial. The headline result is that LLM pipelines cost pennies per trial and take minutes rather than hours, with prompt-chained GPT-4 coming closest to human performance on concept extraction. The author argues this addresses the industrial bottleneck of clinical trials outpacing manual processing, while being explicit that the merged ontology is a categorized concept list because relationships between concepts are dropped.","feed_headline":"LLM pipeline cuts clinical-trial ontology cost from $5 to pennies","feed_subtitle":"On 50 diabetes trials, GPT-4 with prompt chaining nears human concept extraction at a sliver of the time.","key_machinery":"The machinery has two parts: an ontology-generation prompt that restricts the LLM to placing biomarkers, endpoint scores, measurement tools, and questionnaires as subclasses of four fixed classes in OWL ontologies, and a merging stage that processes each clinical-trial ontology triple by triple, using a sorted synonym list to decide whether an entity already exists. The sorted list makes lookup O(log n) and the overall merge O(n), at the cost of discarding all object-property relationships. The evaluation engine is the OQuaRE framework, whose NOCOnto metric—measuring concept-extraction density—serves as the main quality barometer because the merged ontology contains no relations to assess.","core_discovery":"The central claim is that a prompt-based LLM pipeline—using reference material, explicit instructions, a reference ontology template, and optional prompt chaining—can extract biomarkers, endpoint scores, measurement tools, and questionnaires from clinical-trial outcome text and merge them into a single ontology at far lower cost and time than a human, with acceptable quality. The strongest evidence is the extrapolation from 50 trials to roughly 6,200 diabetes trials on the public trial registry: chained GPT-4 would cost about $584 and take about 15 days, versus an estimated $31,000 and 1,550 human hours. On the NOCOnto quality metric, larger and chained models extract more concepts, and chained GPT-4 performs best, while open-weight Llama 3 models offer similar cost and time savings. The paper also proposes a linear-time ontology merging method built on a sorted synonym list, with the explicit drawback that all relationships between concepts are lost during merging.","pith_inferences":["Extending the synonym-list merge to canonicalize relation triples as well as entity triples would preserve relationships while keeping the O(n) scaling; the paper identifies relationships as the gap, so this is a natural next step rather than a claim it makes.","A direct test of medical utility—for instance, using the merged ontology for a relation-dependent task such as matching patient characteristics to trial eligibility criteria—would show whether the dropped relations actually matter; the paper does not run such a test.","Because the NOCOnto metric counts extracted concepts without checking whether they appear in the source trial, an audit that verifies extracted concepts against the trial text would separate true extraction quality from hallucinated concepts.","The methodology's restriction to one disease condition suggests a transfer test on multiple conditions: the fixed four-category template may need to be extended or made condition-dependent before multi-condition trials can be integrated."],"forward_implications":["Extrapolating the measured chained-GPT-4 figures from 50 trials to roughly 6,200 diabetes trials implies the full set could be processed for about $584 in about 15 days, versus an estimated $31,000 and 1,550 hours for a human.","Prompt chaining raises the fraction of syntactically valid, mergeable ontologies; chained GPT-4 jumps from 26% to 86% of trials included.","At per-trial costs of a few cents to a few dollars, real-time ingestion of newly posted trial results becomes economically feasible, so a clinical-trial ontology could be refreshed as trials publish.","The merged ontology is a categorized concept list, so its immediate use is limited to finding which concepts belong to which category; it cannot support reasoning over relationships between concepts.","Open-weight models such as Llama 3 offer the same order-of-magnitude cost and time savings as API models, which matters for deployments that cannot send clinical data to external services."],"supporting_citations":[{"why":"Documents the sheer volume of clinical trials that outruns manual review, which is the problem the automation pipeline targets.","marker":"[4]"},{"why":"Defines the OQuaRE framework and its 19 metrics, the basis for comparing ontology quality across models and the human baseline.","marker":"[12]"},{"why":"Surveys hallucination in language generation, motivating the paper's mitigation of feeding trial outcomes directly into the prompt.","marker":"[17]"},{"why":"Supplies the prompt-engineering tactics (reference material, clear instructions, reference text, chaining) used in the ontology-generation prompts.","marker":"[24]"},{"why":"Provides the automated OQuaRE metrics implementation used to score the generated ontologies.","marker":"[28]"},{"why":"Characterizes GPT-4, one of the four model families compared and the one closest to human performance.","marker":"[8]"},{"why":"Supplies the Llama 3 model description underlying the two open-weight models in the comparison.","marker":"[3]"}],"fun_headline_variants":["LLM chains turn clinical-trial ontologies into a pennies-per-trial job","GPT-4 with prompt chaining nears humans on clinical-trial ontologies at 2% cost","Clinical-trial ontology in days, not months: LLM pipeline delivers","LLMs build clinical-trial ontologies for pennies, near human quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim collapses if a concept list with no relationships between concepts is not a usable ontology for medical practice; the paper itself concedes that relationships are crucial for practical medical applications.","fun_headline_variants_meta":{"raw":{"variants":["LLM chains turn clinical-trial ontologies into a pennies-per-trial job","GPT-4 with prompt chaining nears humans on clinical-trial ontologies at 2% cost","Clinical-trial ontology in days, not months: LLM pipeline delivers","LLMs build clinical-trial ontologies for pennies, near human quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001071,"raw_usage":{"total_tokens":4441,"prompt_tokens":856,"completion_tokens":3585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":3497}},"tokens_in":472,"tokens_out":3585,"duration_ms":25323,"temperature":1.0,"reasoning_tokens":3497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:15:50.672848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Query the merged ontology with a relation-dependent medical question—for example, which measurement tools were used to assess which endpoint scores in the diabetes trials—and observe that the merge step has discarded all object-property assertions, so no answer can be recovered; if a practitioner needs that answer, the claimed viability of the resulting artifact fails.","supporting_citations":[{"cited_title":"Expert Systems with Applica- tions 40(7), 2696–2703 (Jun 2013)","cited_arxiv_id":null,"evidence_quote":"Defines the OQuaRE framework and its 19 metrics, the basis for comparing ontology quality across models and the human baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-engineering tactics (reference material, clear instructions, reference text, chaining) used in the ontology-generation prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the automated OQuaRE metrics implementation used to score the generated ontologies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Characterizes GPT-4, one of the four model families compared and the one closest to human performance."},{"cited_title":"https://llama.meta.com/llama3/, [Accessed 16-06-2024]","cited_arxiv_id":null,"evidence_quote":"Supplies the Llama 3 model description underlying the two open-weight models in the comparison."}],"review_version":1}